Normalization (machine learning)

In machine learning, normalization is a statistical technique with various applications. There are mainly two forms of normalization, data normalization and activation normalization. Data normalization, or feature scaling, is a general technique in statistics, and it includes methods that rescale input data so that they have well-behaved range, mean, variance, and other statistical properties. Activation normalization is specific to deep learning, and it includes methods that rescale the activation of hidden neurons inside a neural network.

Normalization is often used for faster training convergence, less sensitivity to variations in input data, less overfitting, and better generalization to unseen data. They are often theoretically justified as reducing covariance shift, smoother optimization landscapes, increasing regularization, though they are mainly justified by empirical success.

Activation normalization

In a deep neural network, the neural activations are arrays of numbers. The output array of each layer is the input data to the next layer. While this means feature scaling techniques can be applied to normalize neural activations, specialized techniques were developed which were empirically found to work better.

Normalization can also equalize the effect of features to the learning process, preventing data imbalance and make it so that one learning rate works for all the weights.

Typically, the benefits of activation normalization are reported as: stabilizing the gradients during training, less sensitive to weight initialization, faster convergence, larger learning rates, allowing deeper networks to be trained, model regularization, reducing overfitting, better generalization.

Batch normalization

Batch normalization (BatchNorm) operates on the activations of a layer for each mini-batch rather than on the input features across the entire dataset.

It is typically applied after the linear transformation (and before the activation function) within a neural network layer.

Instead of applying a fixed formula like Min-Max or Standardization, BatchNorm learns the optimal scaling and shifting parameters for each layer during training. For a given mini-batch and feature map, it calculates the mean ($\mu _{B}$) and variance ($\sigma _{B}^{2}$) and uses them to normalize the activations. To maintain the expressive power of the network, learnable parameters $\gamma$ (scale) and $\beta$ (shift) are introduced:$${\hat {x_{i}}}={\frac {x_{i}-\mu _{B}}{\sqrt {\sigma _{B}^{2}+\epsilon }}},\quad y_{i}=\gamma {\hat {x_{i}}}+\beta$$

• $x_{i}$ represents individual activations in the mini-batch.
• $\epsilon$ is a small constant added for numerical stability.

It is claimed in the original publication that BatchNorm works by reducing "internal covariance shift", though the claim has both supporters and detractors.

Layer normalization

Layer normalization (LayerNorm) is a common competitor to BatchNorm. Unlike BatchNorm, which normalizes activations across the batch dimension for a given feature, LayerNorm normalizes across all the features within a single data sample. Compared to BatchNorm, LayerNorm's performance is not affected by batch size, making it more stable when using smaller batch sizes or working with recurrent neural networks (RNNs).

It is a key component of Transformers, particularly for natural language processing tasks.

For a given data sample and layer, LayerNorm computes the mean ($\mu$) and variance ($\sigma ^{2}$) over all the features. Similar to BatchNorm, learnable parameters $\gamma$ (scale) and $\beta$ (shift) are applied:$${\hat {x_{i}}}={\frac {x_{i}-\mu }{\sqrt {\sigma ^{2}+\epsilon }}},\quad y_{i}=\gamma {\hat {x_{i}}}+\beta$$

• $x_{i}$ represents the features (activations) for a single data sample.
• $\epsilon$ is a small constant added for numerical stability.

See also

• Data preprocessing
• Feature scaling

Further reading

• "Normalization Layers". labml.ai Deep Learning Paper Implementations. Retrieved 2024-08-07.