User:MatrixHe/sandbox

The MgNet is an abstract and unified mathematical framework proposed by Juncai He and Jinchao Xu that simultaneously recovers some convolutional neural networks (CNN) for image classification and multigrid (MG) methods for solving discretized partial differential equations (PDEs). This model is based on close connections that we have observed and uncovered between the CNN and MG methodologies. For example, pooling operation and feature extraction in CNN correspond directly to restriction operation and iterative smoothers in MG, respectively. As the solution space is often the dual of the data space in PDEs, the analogous concept of feature space and data space (which are dual to each other) is introduced in CNN. With such connections and new concept in the unified model, the function of various convolution operations and pooling used in CNN can be better understood. As a result, modified CNN models (with fewer weights and hyperparameters) are developed that exhibit competitive and sometimes better performance in comparison with existing CNN models when applied to both CIFAR-10 and CIFAR-100 data sets.

Main structure and connections with ResNet

The so-called data and feature space for CNN, which is analogous to the function space and its duality in the theory of multigrid methods is introduced o examine further connections between CNN and multigrid. With this new concept for CNN, MgNet and a further research proposes the constrained data-feature mapping model in every grid as

$A\ast u=f,$

where $f$ belongs to the data space and $u$ belongs to the feature space such that

$u\geq 0$.

The feature extraction process can then be obtained through an iterative procedure for solving the above system in each grids. For example, if we apply a single step residual correction step for the above system, we have

$u^{i}=u^{i-1}+\sigma \circ B^{i}\ast \sigma (f^{}-A\ast u^{i-1}),\quad i=1:\nu ,$

with $u\approx u^{\nu }$.

If we consider the residual of the above iterative $r^{i}=f-A\ast u^{i}$, we have

$r^{i}=r^{i-1}-A\ast \sigma \circ B^{i}\ast \sigma (r^{i-1}),\quad i=1:\nu .$

This is exact the basic block scheme in Pre-act ResNet. For further connections, if

Connections with other CNN architectures

The above iterative scheme can be interpreted as both the feature extraction step in ResNet type models and the smoothing step in multigrid method. Under this framework, several successful CNN architectures can be understood as different smoothing steps for example:

CNN architectures	Smoothing methods in MG
ResNet	Single step residual correction
DenseNet	Multi-step residual correction
LM-ResNet	Chebyshev-semi residual correction

Summary

Different with the dynamic system viewpoint, the MgNet framework opens a new door to the mathematical understanding, analysis and improvements of deep learning models. The very preliminary results presented in have demonstrated the great potential of MgNet from both theoretical and practical viewpoints. Obviously many aspects of MgNet should be further explored and expect to be much improved. In fact, only very few techniques from multigrid method have been tried in and many more in-depth techniques from multigrid require further study for deep neural networks, especially CNN. In particular, it is believed that the MgNet framework will lead to improved CNN that only has a small fraction of the number of weights that are required by the current CNN. On the other hand, the techniques in CNN can also be used to develop new generation of multigrid and especially algebraic multigrid methods for solving partial differential equations.