Generalized Hebbian algorithm

The generalized Hebbian algorithm (GHA), also known in the literature as Sanger's rule, is a linear feedforward neural network for unsupervised learning with applications primarily in principal components analysis. First defined in 1989, it is similar to Oja's rule in its formulation and stability, except it can be applied to networks with multiple outputs. The name originates because of the similarity between the algorithm and a hypothesis made by Donald Hebb about the way in which synaptic strengths in the brain are modified in response to experience, i.e., that changes are proportional to the correlation between the firing of pre- and post-synaptic neurons.

Theory

Consider a problem of learning a linear code for some data. Each data is a multi-dimensional vector $x\in \mathbb {R} ^{n}$, and can be (approximately) represented as a linear sum of linear code vectors $w_{1},\dots ,w_{m}$. When $m=n$, it is possible to exactly represent the data. If $m<n$, it is possible to approximately represent the data. To minimize the L2 loss of representation, $w_{1},\dots ,w_{m}$ should be the highest principal component vectors.

The GHA is an iterative method to learn the highest principal component vectors in a form that resembles unsupervised Hebbian learning in neural networks.

Consider a one-layered neural network with $n$ input neurons and $m$ output neurons $y_{1},\dots ,y_{m}$. The linear code vectors are the connection strengths, that is, $w_{ij}$ is the synaptic weight or connection strength between the $j$-th input and $i$-th output neurons.

The GHA learning rule is of the form

$\,\Delta w_{ij}~=~\eta \left(y_{i}x_{j}-y_{i}\sum _{k=1}^{i}w_{kj}y_{k}\right)$

where $\eta$ is the learning rate parameter.

Oja's rule is the special case where $m=1$.

Derivation

In matrix form, Oja's rule can be written

$\,{\frac {{\text{d}}w(t)}{{\text{d}}t}}~=~w(t)Q-\mathrm {diag} [w(t)Qw(t)^{\mathrm {T} }]w(t)$,

and the Gram-Schmidt algorithm is

$\,\Delta w(t)~=~-\mathrm {lower} [w(t)w(t)^{\mathrm {T} }]w(t)$,

where w(t) is any matrix, in this case representing synaptic weights, Q = η x x^T is the autocorrelation matrix, simply the outer product of inputs, diag is the function that diagonalizes a matrix, and lower is the function that sets all matrix elements on or above the diagonal equal to 0. We can combine these equations to get our original rule in matrix form,

$\,\Delta w(t)~=~\eta (t)\left(\mathbf {y} (t)\mathbf {x} (t)^{\mathrm {T} }-\mathrm {LT} [\mathbf {y} (t)\mathbf {y} (t)^{\mathrm {T} }]w(t)\right)$,

where the function LT sets all matrix elements above the diagonal equal to 0, and note that our output y(t) = w(t) x(t) is a linear neuron.

Applications

The GHA is used in applications where a self-organizing map is necessary, or where a feature or principal components analysis can be used. Examples of such cases include artificial intelligence and speech and image processing.

Its importance comes from the fact that learning is a single-layer process—that is, a synaptic weight changes only depending on the response of the inputs and outputs of that layer, thus avoiding the multi-layer dependence associated with the backpropagation algorithm. It also has a simple and predictable trade-off between learning speed and accuracy of convergence as set by the learning rate parameter η.

Features learned by GHA running on 8-by-8 patches of Caltech 101.

Features found by Principal Component Analysis on the same Caltech 101 dataset.

As an example, (Olshausen and Field, 1996) performed GHA on 8-by-8 patches of photos of natural scenes, and found that it results in Fourier-like features. The features are the same as the principal components found by PCA, as expected, and that, the features are determined by the $64\times 64$ variance matrix of the samples of 8-by-8 patches. In other words, it is determined by the second-order statistics of the pixels in images. They criticized this as insufficient to capture higher-order statistics which are necessary to explain the Gabor-like features of simple cells in the primary visual cortex.

See also

• Hebbian learning
• Factor analysis
• Contrastive Hebbian learning
• Oja's rule