Neural style transfer explained

Neural style transfer (NST) refers to a class of software algorithms that manipulate digital images, or videos, in order to adopt the appearance or visual style of another image. NST algorithms are characterized by their use of deep neural networks for the sake of image transformation. Common uses for NST are the creation of artificial artwork from photographs, for example by transferring the appearance of famous paintings to user-supplied photographs. Several notable mobile apps use NST techniques for this purpose, including DeepArt and Prisma. This method has been used by artists and designers around the globe to develop new artwork based on existent style(s).

History

NST is an example of image stylization, a problem studied for over two decades within the field of non-photorealistic rendering. The first two example-based style transfer algorithms were image analogies^[1] and image quilting.^[2] Both of these methods were based on patch-based texture synthesis algorithms.

Given a training pair of images–a photo and an artwork depicting that photo–a transformation could be learned and then applied to create new artwork from a new photo, by analogy. If no training photo was available, it would need to be produced by processing the input artwork; image quilting did not require this processing step, though it was demonstrated on only one style.

NST was first published in the paper "A Neural Algorithm of Artistic Style" by Leon Gatys et al., originally released to ArXiv 2015,^[3] and subsequently accepted by the peer-reviewed CVPR conference in 2016.^[4] The original paper used a VGG-19 architecture^[5] that has been pre-trained to perform object recognition using the ImageNet dataset.

In 2017, Google AI introduced a method^[6] that allows a single deep convolutional style transfer network to learn multiple styles at the same time. This algorithm permits style interpolation in real-time, even when done on video media.

Mathematics

This section closely follows the original paper.

Overview

The idea of Neural Style Transfer (NST) is to take two images—a content image

and a style image —and generate a third image that minimizes a weighted combination of two loss functions: a content loss and a style loss .

The total loss is a linear sum of the two:$$\backslash mathcal\_(\backslash vec,\; \backslash vec\; a,\; \backslash vec)\; =\; \backslash alpha\; \backslash mathcal\_(\backslash vec,\; \backslash vec)\; +\; \backslash beta\; \backslash mathcal\; L\_(\backslash vec,\; \backslash vec)$$By jointly minimizing the content and style losses, NST generates an image that blends the content of the content image with the style of the style image.

Both the content loss and the style loss measures the similarity of two images. The content similarity is the weighted sum of squared-differences between the neural activations of a single convolutional neural network (CNN) on two images. The style similarity is the weighted sum of Gram matrices within each layer (see below for details).

The original paper used a VGG-19 CNN, but the method works for any CNN.

Symbols

Let $\backslash vec$ be an image input to a CNN.

Let $F^l\; \backslash in\; \backslash mathbb^$ be the matrix of filter responses in layer $l$ to the image $\backslash vec$, where:

• $N\_l$ is the number of filters in layer $l$ ;
• $M\_l$ is the height times the width (i.e. number of pixels) of each filter in layer $l$ ;
• $F\_^l(\backslash vec\; x)$ is the activation of the $i^$ filter at position $j$ in layer $l$.

A given input image $\backslash vec$ is encoded in each layer of the CNN by the filter responses to that image, with higher layers encoding more global features, but losing details on local features.

Content loss

Let $\backslash vec$ be an original image. Let $\backslash vec$ be an image that is generated to match the content of $\backslash vec$. Let $P^l$ be the matrix of filter responses in layer $l$ to the image $\backslash vec$.

The content loss is defined as the squared-error loss between the feature representations of the generated image and the content image at a chosen layer

of a CNN:$$\backslash mathcal\_(\backslash vec,\; \backslash vec,\; l)=\backslash frac\; \backslash sum\_\backslash left(A\_^l(\backslash vec\; x)-\; A\_^l(\backslash vec\; p)\backslash right)^2$$where and are the activations of the filter at position in layer for the generated and content images, respectively. Minimizing this loss encourages the generated image to have similar content to the content image, as captured by the feature activations in the chosen layer.

The total content loss is a linear sum of the content losses of each layer:

, where the are positive real numbers chosen as hyperparameters.

Style loss

The style loss is based on the Gram matrices of the generated and style images, which capture the correlations between different filter responses at different layers of the CNN:$$\backslash mathcal\_(\backslash vec,\; \backslash vec)=\backslash sum\_^L\; w\_l\; E\_l,$$where$$E\_l=\backslash frac\; \backslash sum\_\backslash left(G\_^l(\backslash vec\; x)-G\_^l(\backslash vec\; a)\backslash right)^2.$$Here,

and are the entries of the Gram matrices for the generated and style images at layer . Explicitly, $$G\_^l(\backslash vec\; x)=\backslash sum\_k\; F\_^l(\backslash vec\; x)\; F\_^l(\backslash vec\; x)$$

Minimizing this loss encourages the generated image to have similar style characteristics to the style image, as captured by the correlations between feature responses in each layer. The idea is that activation pattern correlations between filters in a single layer captures the "style" on the order of the receptive fields at that layer.

Similarly to the previous case, the

are positive real numbers chosen as hyperparameters.

Hyperparameters

In the original paper, they used a particular choice of hyperparameters.

The style loss is computed by

for the outputs of layers conv1_1, conv2_1, conv3_1, conv4_1, conv5_1 in the VGG-19 network, and zero otherwise. The content loss is computed by for conv4_2, and zero otherwise.

The ratio

.

Training

Image

is initially approximated by adding a small amount of white noise to input image and feeding it through the CNN. Then we successively backpropagate this loss through the network with the CNN weights fixed in order to update the pixels of . After several thousand epochs of training, an (hopefully) emerges that matches the style of and the content of .

, when implemented on a GPU, it takes a few minutes to converge.

Extensions

In some practical implementations, it is noted that the resulting image has too much high-frequency artifact, which can be suppressed by adding the total variation to the total loss.^[7]

Compared to VGGNet, AlexNet does not work well for neural style transfer.^[8]

NST has also been extended to videos.^[9]

Subsequent work improved the speed of NST for images by using special-purpose normalizations.^{[10] [11]}

In a paper by Fei-Fei Li et al. adopted a different regularized loss metric and accelerated method for training to produce results in real-time (three orders of magnitude faster than Gatys).^[12] Their idea was to use not the pixel-based loss defined above but rather a 'perceptual loss' measuring the differences between higher-level layers within the CNN. They used a symmetric convolution-deconvolution CNN. Training uses a similar loss function to the basic NST method but also regularizes the output for smoothness using a total variation (TV) loss. Once trained, the network may be used to transform an image into the style used during training, using a single feed-forward pass of the network. However the network is restricted to the single style in which it has been trained.^[12]

In a work by Chen Dongdong et al. they explored the fusion of optical flow information into feedforward networks in order to improve the temporal coherence of the output.^[13]

Most recently, feature transform based NST methods have been explored for fast stylization that are not coupled to single specific style and enable user-controllable blending of styles, for example the whitening and coloring transform (WCT).^[14]

Notes and References

1. Book: Hertzmann . Aaron . Jacobs . Charles E. . Oliver . Nuria . Curless . Brian . Salesin . David H. . Image analogies . August 2001 . Proceedings of the 28th annual conference on Computer graphics and interactive techniques . https://dl.acm.org/doi/10.1145/383259.383295 . ACM . 327–340 . 10.1145/383259.383295 . 978-1-58113-374-5.
2. Book: Efros . Alexei A. . Freeman . William T. . Image quilting for texture synthesis and transfer . August 2001 . Proceedings of the 28th annual conference on Computer graphics and interactive techniques . https://dl.acm.org/doi/10.1145/383259.383296 . ACM . 341–346 . 10.1145/383259.383296 . 978-1-58113-374-5.
3. 1508.06576. A Neural Algorithm of Artistic Style. Leon A.. Gatys. Alexander S.. Ecker. Matthias. Bethge. 26 August 2015. cs.CV.
4. Gatys . Leon A. . Ecker . Alexander S. . Bethge . Matthias . 2016 . Image Style Transfer Using Convolutional Neural Networks . IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . 2414–2423.
5. Web site: Very Deep CNNS for Large-Scale Visual Recognition. 2014. Robots.ox.ac.uk. 13 February 2019.
6. 1610.07629. A Learned Representation for Artistic Style. Vincent. Dumoulin. Jonathon S.. Shlens. Manjunath. Kudlur. 9 February 2017. cs.CV.
7. Jing . Yongcheng . Yang . Yezhou . Feng . Zunlei . Ye . Jingwen . Yu . Yizhou . Song . Mingli . 2020-11-01 . Neural Style Transfer: A Review . IEEE Transactions on Visualization and Computer Graphics . 26 . 11 . 3365–3385 . 10.1109/TVCG.2019.2921336 . 31180860 . 1077-2626. 1705.04058 .
8. Web site: Neural Style transfer with Deep Learning Dawars' blog . 2024-09-23 . dawars.me.
9. Book: 1604.08610. Ruder. Manuel. Pattern Recognition. Dosovitskiy. Alexey. Brox. Thomas. Artistic Style Transfer for Videos. Lecture Notes in Computer Science. 2016. 9796. 26–36. 10.1007/978-3-319-45886-1_3. 978-3-319-45885-4. 47476652.
10. Ulyanov . Dmitry . Instance Normalization: The Missing Ingredient for Fast Stylization . 2017-11-06 . 1607.08022 . Vedaldi . Andrea . Lempitsky . Victor. cs.CV .
11. Huang . Xun . Belongie . Serge . 2017 . Arbitrary Style Transfer in Real-Time With Adaptive Instance Normalization . 1501–1510. 1703.06868 .
12. 1603.08155. Johnson. Justin. Perceptual Losses for Real-Time Style Transfer and Super-Resolution. Alahi. Alexandre. Li. Fei-Fei. cs.CV. 2016.
13. 1703.09211. Chen. Dongdong. Coherent Online Video Style Transfer. Liao. Jing. Yuan. Lu. Yu. Nenghai. Hua. Gang. cs.CV. 2017.
14. 1705.08086. Li. Yijun. Universal Style Transfer via Feature Transforms. Fang. Chen. Yang. Jimei. Wang. Zhaowen. Lu. Xin. Yang. Ming-Hsuan. cs.CV. 2017.