Neural style transfer (NST) refers to a class of software algorithms that manipulate digital images, or videos, in order to adopt the appearance or visual style of another image. NST algorithms are characterized by their use of deep neural networks for the sake of image transformation. Common uses for NST are the creation of artificial artwork from photographs, for example by transferring the appearance of famous paintings to user-supplied photographs. Several notable mobile apps use NST techniques for this purpose, including DeepArt and Prisma. This method has been used by artists and designers around the globe to develop new artwork based on existent style(s).
NST is an example of image stylization, a problem studied for over two decades within the field of non-photorealistic rendering. The first two example-based style transfer algorithms were image analogies[1] and image quilting.[2] Both of these methods were based on patch-based texture synthesis algorithms.
Given a training pair of images–a photo and an artwork depicting that photo–a transformation could be learned and then applied to create new artwork from a new photo, by analogy. If no training photo was available, it would need to be produced by processing the input artwork; image quilting did not require this processing step, though it was demonstrated on only one style.
NST was first published in the paper "A Neural Algorithm of Artistic Style" by Leon Gatys et al., originally released to ArXiv 2015,[3] and subsequently accepted by the peer-reviewed CVPR conference in 2016.[4] The original paper used a VGG-19 architecture[5] that has been pre-trained to perform object recognition using the ImageNet dataset.
In 2017, Google AI introduced a method[6] that allows a single deep convolutional style transfer network to learn multiple styles at the same time. This algorithm permits style interpolation in real-time, even when done on video media.
The process of NST assumes an input image
p
a
The image
p
C(p)
p
The style image
a
S(a)
a
The goal of NST is to synthesize an output image
x
p
a
C(x)=C(p)
S(x)=S(a)
An iterative optimization (usually gradient descent) then gradually updates
x
l{L(x)}=|C(x)-C(p)|+k|S(x)-S(a)|
where
|.|
k
Image
x
p
x
x
a
p
Algorithms are typically implemented for GPUs, so that training takes a few minutes.
NST has also been extended to videos.[7]
Subsequent work improved the speed of NST for images.
In a paper by Fei-Fei Li et al. adopted a different regularized loss metric and accelerated method for training to produce results in real-time (three orders of magnitude faster than Gatys).[8] Their idea was to use not the pixel-based loss defined above but rather a 'perceptual loss' measuring the differences between higher-level layers within the CNN. They used a symmetric encoder-decoder CNN. Training uses a similar loss function to the basic NST method but also regularizes the output for smoothness using a total variation (TV) loss. Once trained, the network may be used to transform an image into the style used during training, using a single feed-forward pass of the network. However the network is restricted to the single style in which it has been trained.[9]
In a work by Chen Dongdong et al. they explored the fusion of optical flow information into feedforward networks in order to improve the temporal coherence of the output.[10]
Most recently, feature transform based NST methods have been explored for fast stylization that are not coupled to single specific style and enable user-controllable blending of styles, for example the whitening and coloring transform (WCT).[11]