Model compression

Model compression refers techniques in machine learning to reduce the size of trained models. Large models can achieve high accuracy but often come at the cost of significant resource requirements. Compression techniques aim to compress models without significantly sacrificing performance. Smaller models require less storage space and consume less memory and compute during inference.

Compressed models enable deployment on resource-constrained devices like smartphones and embedded systems, on-device AI, edge computing, and consumer electronics computers. Efficient inference is also valuable for large corporations who serve large model inference over an API, allowing them to reduce computational costs and improve response times for users.

Model compression is not model distillation, which trains a separate smaller model that imitates the input-output behavior of the larger model.

Techniques

Several techniques are employed for model compression.

Pruning

Pruning sparsifies a large model by setting some parameters to exactly zero. This effectively reduces the number of parameters. This allows the use of sparse matrix operations, which are faster than dense matrix operations.

Pruning criteria can be based on magnitudes of parameters, the statistical pattern of neural activations, Hessian values, etc.

Quantization

Quantization reduces the numerical precision of weights and activations. For example, instead of storing weights as 32-bit floating-point numbers, they can be represented using 8-bit integers. Low-precision parameters take up less space, and takes less compute to perform arithmetics with.

It is also possible to quantize some parameters more aggressively than others, so for example, a less important parameter can have 8-bit precision while another, more important parameter, can have 16-bit precision. Inference with such models requires mixed-precision arithmetics.

Low-rank factorization

Main article: Low-rank approximation

Weight matrices can be approximated by low-rank matrices. Let $W$ be a weight matrix of shape $m\times n$. A low-rank approximation is $W\approx UV^{T}$, where $U$ and $V$ are matrices of shapes $m\times k,n\times k$. When $k$ is small, this both reduces the number of parameters needed to represent $W$ approximately, and accelerates matrix multiplication by $W$.

Low-rank approximations can be found by singular value decomposition (SVD).

Training

Model compression is usually decoupled from training, that is, a model is first trained without regard for how it might be compressed, then it is compressed. However, it is possible to combine model compression with training.

The "train big, then compress" method trains a large model for a small number of training steps, then heavily compress the model. It is found that at the same compute budget, this method results in a better model than lightly compressed, small models.

References

• Review papers
  • Li, Zhuo; Li, Hengyi; Meng, Lin (March 12, 2023). "Model Compression for Deep Neural Networks: A Survey". Computers. 12 (3). MDPI AG: 60. doi:10.3390/computers12030060. ISSN 2073-431X.
  • "Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey". IEEE Xplore. March 20, 2020. Retrieved October 18, 2024.
  • Cheng, Yu; Wang, Duo; Zhou, Pan; Zhang, Tao (October 23, 2017). "A Survey of Model Compression and Acceleration for Deep Neural Networks". arXiv.org. Retrieved October 18, 2024.
  • Choudhary, Tejalal; Mishra, Vipul; Goswami, Anurag; Sarangapani, Jagannathan (February 8, 2020). "A comprehensive survey on model compression and acceleration". Artificial Intelligence Review. 53 (7). Springer Science and Business Media LLC: 5113–5155. doi:10.1007/s10462-020-09816-7. ISSN 0269-2821.