Deep image prior is a type of convolutional neural network used to enhance a given image with no prior training data other than the image itself.A neural network is randomly initialized and used as prior to solve inverse problems such as noise reduction, super-resolution, and inpainting. Image statistics are captured by the structure of a convolutional image generator rather than by any previously learned capabilities.
x*=minxE(x;x0)+R(x)
x
x0
E(x;x0)
Deep neural networks learn a generator/decoder
x=f\theta(z)
z
x
The image corruption method used to generate
x0
In this approach, the
R(x)
R(x)=0
R(x)=+infin
\theta*=argmin\thetaE(f\theta(z);x0)
x*=
| f | |
| \theta* |
(z)
The minimizer
\theta*
x*
A parameter θ may be used to recover any image, including its noise. However, the network is reluctant to pick up noise because it contains high impedance while useful signal offers low impedance. This results in the θ parameter approaching a good-looking local optimum so long as the number of iterations in the optimization process remains low enough not to overfit data.
Typically, the deep neural network model for deep image prior uses a U-Net like model without the skip connections that connect the encoder blocks with the decoder blocks. The authors in their paper mention that "Our findings here (and in other similar comparisons) seem to suggest that having deeper architecture is beneficial, and that having skip-connections that work so well for recognition tasks (such as semantic segmentation) is highly detrimental."[1]
The principle of denoising is to recover an image
x
x0
x0=x+\epsilon
\epsilon
The quadratic energy function
E(x,x0)=||x-x
| 2 | |
| 0|| |
\theta*
min\theta||f\theta
| 2 | |
| (z)-x | |
| 0|| |
Super-resolution is used to generate a higher resolution version of image x. The data term is set to
E(x;x0)=||d(x)-x
| 2 | |
| 0|| |
Inpainting is used to reconstruct a missing area in an image
x0
m\in\{0,1\}H
E(x;x0)=||(x-x0)\odotm||2
\odot
The intuition behind this is that the loss is computed only on the known pixels in the image, and the network is going to learn enough about the image to fill in unknown parts of the image even though the computed loss doesn't include those pixels. This strategy is used to remove image watermarks by treating the watermark as missing pixels in the image.
This approach may be extended to multiple images. A straightforward example mentioned by the author is the reconstruction of an image to obtain natural light and clarity from a flash–no-flash pair. Video reconstruction is possible but it requires optimizations to take into account the spatial differences.
See Astronomy Picture of the Day (APOD) of 2024-02-18[4]