Deep image prior

Deep Image Prior is a type of convolutional neural network used to enhance a given image with no prior training data other than the image itself. A neural-network is randomly initialized, and used as prior to solve inverse problems such as noise reduction, super-resolution, and inpainting. Image statistics is captured by the structure of a convolutional image generator rather than by any previously learned capabilities.

Method

Background

Inverse problems such as noise reduction, super-resolution, and inpainting can be formulated as the optimization task $x^{*}=min_{x}E(x;x_{0})+R(x)$, where $x$ is an image, $x_{0}$ a corrupted representation of that image, $E(x;x_{0})$ is a task-dependent data term, and R(x) is the regularizer. This forms an energy minimization problem.

Deep neural networks learn a generator/decoder $x=f_{\theta }(z)$ which maps a random code vector $z$ to an image $x$.

The image corruption method used to generate $x_{0}$ is selected for the specific application.

Specifics

In this approach, the $R(x)$ prior is replaced with the implicit prior captured by the neural network (where $R(x)=0$ for images that can be produced by a deep neural networks and $R(x)=+\infty$ otherwise). This yields the equation for the minimizer $\theta ^{*}=argmin_{\theta }E(f_{\theta }(z);x_{0})$ and the result of the optimization process $x^{*}=f_{\theta ^{*}}(z)$.

The minimizer $\theta ^{*}$ (typically a gradient descent) starts from a randomly initialized parameters and descends into a local best result to yield the $x^{*}$ restoration function.

Overfitting

A parameter θ may be used to recover any image, including its noise. However, the network is reluctant to pick up noise because it contains high impedance while useful signal offers low impedance. This results in the θ parameter approaching a good-looking local optimum so long as the number of iterations in the optimization process remains low enough not to overfit data.

Implementations

• A reference implementation written in Python 2 with the PyTorch library was released by the author under the Apache 2.0 license: deep-image-prior
• A Tensorflow-based implementation written in Python 2 and released under the CC-SA 3.0 license: deep-image-prior-tensorflow
• A Keras-based implementation written in Python 2 and released under the GPLv3: machine_learning_denoising

References

Ulyanov, Dmitry; Vedaldi, Andrea; Lempitsky, Victor (30 November 2017). "Deep Image Prior". arXiv:1711.10925v2 [Vision and Pattern Recognition Computer Vision and Pattern Recognition].