User:MatrixHe/sandbox

The MgNet is an abstract and unified mathematical framework which simultaneously recovers some ResNet type convolutional neural networks (CNNs) and multigrid methods for solving discretized partial differential equations (PDEs). MgNet can be obtained by making some very minor modifications of a classic geometric multigrid method. Actually, connections between ResNet and classical multigrid methods were acknowledged in the original paper of ResNet from the viewpoint how residuals are applied in both methods.  MgNet makes such a connection more direct and more clear and it makes it possible to directly obtain a class of efficient CNN models by simply making some very minor modification of a typical multigrid cycle but keeping the identically same algorithm structure.

Main structure and connections with ResNet

The so-called data and feature space for CNN, which is analogous to the function space and its duality in the theory of multigrid methods is introduced o examine further connections between CNN and multigrid. With this new concept for CNN, MgNet and a further research proposes the constrained data-feature mapping model in every grid as

$A\ast u=f,$

where $f$ belongs to the data space and $u$ belongs to the feature space such that

$u\geq 0$.

The feature extraction process can then be obtained through an iterative procedure for solving the above system in each grids. For example, if we apply a single step residual correction step for the above system, we have

$u^{i}=u^{i-1}+\sigma \circ B^{i}\ast \sigma (f^{}-A\ast u^{i-1}),\quad i=1:\nu ,$

with $u\approx u^{\nu }$.

If we consider the residual of the above iterative $r^{i}=f-A\ast u^{i}$, we have

$r^{i}=r^{i-1}-A\ast \sigma \circ B^{i}\ast \sigma (r^{i-1}),\quad i=1:\nu .$

This is exact the basic block scheme in Pre-act ResNet.

Connections with other CNN architectures

The above iterative scheme can be interpreted as both the feature extraction step in ResNet type models and the smoothing step in multigrid method. Under this framework, several successful CNN architectures can be understood as different smoothing steps for example:

CNN architectures	Smoothing methods in MG
ResNet	Single step residual correction
DenseNet	Multi-step residual correction
LM-ResNet	Chebyshev-semi residual correction

Summary

Different with the dynamic system viewpoint, the MgNet framework opens a new door to the mathematical understanding, analysis and improvements of deep learning models. The very preliminary results presented in have demonstrated the great potential of MgNet from both theoretical and practical viewpoints. Obviously many aspects of MgNet should be further explored and expect to be much improved. In fact, only very few techniques from multigrid method have been tried in and many more in-depth techniques from multigrid require further study for deep neural networks, especially CNN. In particular, it is believed that the MgNet framework will lead to improved CNN that only has a small fraction of the number of weights that are required by the current CNN. On the other hand, the techniques in CNN can also be used to develop new generation of multigrid and especially algebraic multigrid methods for solving partial differential equations.