Pruning (artificial neural network)

In the context of artificial neural network, pruning is the practice of removing parameters (which may entail removing individual parameters, or parameters in groups such as by neurons) from an existing network. The goal of this process is to maintain accuracy of the network while increasing its efficiency. This can be done to reduce the computational resources required to run the neural network. After a network is trained to a desired solution with the training data, units (hidden layer nodes or interconnections) are analysed to determine which are not contributing to the solution. There are several approaches described in the literature to determine non-contributing units. A widely used approach is that non-essential units can be determined by a form of sensitivity analysis.So far, few studies have been performed in the analysis of pruning algorithms for classification of remotely-sensed data.

A basic algorithm for pruning is as follows:

1. Evaluate the importance of each neuron.
2. Rank the neurons according to their importance (assuming there is a clearly defined measure for "importance").
3. Remove the least important neuron.
4. Check a termination condition (to be determined by the user) to see whether to continue pruning.

Types

Magnitude based Pruning

The Magnitude Based pruning technique (MB) is the simplest pruning algorithm . It is based on deleting interconnections with small ‘saliency’, i.e. those whose deletion will have the least effect on the training error. Saliency corresponds to the magnitude (weight) value of the interconnections. It is assumed that the interconnections whose magnitude value is small will have minor effect on the performance of the network. After reasonable initial training, the interconnection having the smallest magnitude value is removed. The network is then retrained and the process is repeated in an iterative fashion until the training error reaches a certain limit.

Optimum Brain Damage

The Optimum Brain Damage pruning algorithm (OBD), introduced by Le Cun, Denker and Solla in 1990, is based on second order derivatives of the error function. The aim is to iteratively delete the weights whose deletion will result in the least increase of the error in the network. There is an important problem in the estimation of formula, is the size of the Hessian matrix. The calculation of the Hessian matrix is time-consuming, hence Le Cun et al. (1990) assume that the Hessian is diagonal. On the other hand, Hassibi and Stork (1993) argue that Hessian for every problem they have considered are strongly non-diagonal, and this leads OBD to eliminate the wrong weights.

Optimum Brain Surgeon pruning algorithm

The Optimum Brain Surgeon pruning algorithm (OBS), introduced by Hassibi and Stork in 1993), is a more complex form of Optimum Brain Damage (OBD). Although OBS and OBD are basically based on the same theoretical approach, OBS does not make any assumption about the form ofHessian matrix. Therefore, OBS is more complex and robust than OBD. It is claimed that optimum brain surgeon (OBS) is significantly better than magnitude based (MB) and optimal brain damage(OBD) techniques, and OBS permits the pruning of more weights than other methods for the same error on the training set, and thus yields better generalization on test data. The most important problem with OBS is that the inverse of the Hessian matrix has to be computed to judge saliency and weight change for every link. Therefore, this method is quite slow and takes much memory compared to the other methods.