Fréchet inception distance explained

The Fréchet inception distance (FID) is a metric used to assess the quality of images created by a generative model, like a generative adversarial network (GAN)^[1] or a diffusion model.^[2]

The FID compares the distribution of generated images with the distribution of a set of real images (a "ground truth" set). Rather than comparing individual images, mean and covariance statistics of many images generated by the model are compared with the same statistics generated from images in the ground truth or reference set. A convolutional neural network such as an inception architecture is used to produce higher-level features describing the images, thus leading to the name Fréchet inception distance.

The FID is inspired by the earlier inception score (IS) metric which evaluates only the distribution of generated images. The FID metric does not replace the IS metric; classifiers that achieve the best (lowest) FID score tend to have greater sample variety while classifiers achieving the best (highest) IS score tend to have better quality within individual images.^[3]

The FID metric was introduced in 2017, and is the current standard metric for assessing the quality of models that generate synthetic images as of 2024. It has been used to measure the quality of many recent models including the high-resolution StyleGAN1^[4] and StyleGAN2^[5] networks, and diffusion models.

The FID attempts to compare images visually through deep layers of an inception network. More recent works take this further by instead comparing CLIP embeddings of the images.^{[6] [7]}

Overview

The purpose of the FID score is to measure the diversity of images created by a generative model with images in a reference dataset. The reference dataset could be ImageNet or COCO-2014.^[8] Using a large dataset as a reference is important as the reference image set should represent the full diversity of images which the model attempts to create.

Generative models such as diffusion models produce novel images that have features from the reference set, but are themselves quite different from any image in the training set. So the quality of these models cannot be assessed by simply comparing each image to an image in the training set pixel-by-pixel, as done, for example, with the L2 norm.

and . The distance between the two distributions is calculated as the earth mover's distance or the Wasserstein distance between the two Gaussian distributions.

Rather than directly comparing images pixel by pixel (for example, as done by the L2 norm), the FID compares the mean and standard deviation of the deepest layer in Inception v3 (the 2048-dimensional activation vector of its last pooling layer.) These layers are closer to output nodes that correspond to real-world objects such as a specific breed of dog or an airplane, and further from the shallow layers near the input image. So the FID compares how often the same high-level features are found within the two sets of images.After every image has been processed through the inception architecture, the means and covariances of the activation of the last layer on the two datasets are compared with the distance$$d\_(\backslash mathcal\; N(\backslash mu,\; \backslash Sigma),\; \backslash mathcal\; N(\backslash mu\text{'},\; \backslash Sigma\text{'}))^2\; =\; \backslash lVert\; \backslash mu\; -\; \backslash mu\text{'}\; \backslash rVert^2\_2\; +\; \backslash operatorname\backslash left(\backslash Sigma\; +\; \backslash Sigma\text{'}\; -2\backslash left(\backslash Sigma\; \backslash Sigma\text{'}\; \backslash right)^\backslash frac\; \backslash right)$$Higher distances indicate a poorer generative model. A score of 0 indicates a perfect model.

Formal Definition

For any two probability distributions

over having finite mean and variances, their earth-mover's distance or Fréchet distance is^[9] $$d\_F\; (\backslash mu,\; \backslash nu):=\backslash left(\backslash inf\_\; \backslash int\_\; \backslash |x-y\backslash |^2\; \backslash ,\; \backslash mathrm\; \backslash gamma\; (x,\; y)\; \backslash right)^,$$where is the set of all measures on with marginals

\mu

and

\nu

on the first and second factors respectively. (The set is also called the set of all couplings of

\mu

and

\nu

.). and , it is expressed in closed form as^[10] $$d\_(\backslash mathcal\; N(\backslash mu,\; \backslash Sigma),\; \backslash mathcal\; N(\backslash mu\text{'},\; \backslash Sigma\text{'}))^2\; =\; \backslash lVert\; \backslash mu\; -\; \backslash mu\text{'}\; \backslash rVert^2\_2\; +\; \backslash operatorname\backslash left(\backslash Sigma\; +\; \backslash Sigma\text{'}\; -2\backslash left(\backslash Sigma\; \backslash Sigma\text{'}\; \backslash right)^\backslash frac\; \backslash right)$$This allows us to define the FID in pseudocode form:

INPUT a function

f:\Omega_X\to\Rⁿ

.

INPUT two datasets

S,S'\subset\Omega_X

.

Compute

f(S),f(S')\subset\Rⁿ

.

Fit two gaussian distributions

lN(\mu,\Sigma),lN(\mu',\Sigma')

, respectively for

f(S),f(S')

.

RETURN

d_F(lN(\mu,\Sigma),lN(\mu',\Sigma'))²

.

In most practical uses of the FID, is the space of images, and is an Inception v3 model trained on the ImageNet, but without its final classification layer. Technically, it is the 2048-dimensional activation vector of its last pooling layer. Of the two datasets , one of them is a reference dataset, which could be the ImageNet itself, and the other is a set of images generated by a generative model, such as GAN, or diffusion model.

Variants

Specialized variants of FID have been suggested as evaluation metric for music enhancement algorithms as Fréchet Audio Distance (FAD),^[11] for generative models of video as Fréchet Video Distance (FVD),^[12] and for AI-generated molecules as Fréchet ChemNet Distance (FCD).^[13]

Limitations

Chong and Forsyth^[14] showed FID to be statistically biased, in the sense that their expected value over a finite data is not their true value. Also, because FID measured the Wasserstein distance towards the ground-truth distribution, it is inadequate for evaluating the quality of generators in domain adaptation setups, or in zero-shot generation. Finally, while FID is more consistent with human judgement than previously used inception score, there are cases where FID is inconsistent with human judgment (e.g. Figure 3,5 in Liu et al.).^[15]

See also

• Fréchet distance

Notes and References

1. Heusel . Martin . Ramsauer . Hubert . Unterthiner . Thomas . Nessler . Bernhard . Hochreiter . Sepp . GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium . Advances in Neural Information Processing Systems . 2017 . 30 . 1706.08500 . en.
2. Esser . Patrick . Scaling Rectified Flow Transformers for High-Resolution Image Synthesis . 2024-03-05 . 2403.03206 . Kulal . Sumith . Blattmann . Andreas . Entezari . Rahim . Müller . Jonas . Saini . Harry . Levi . Yam . Lorenz . Dominik . Sauer . Axel. cs.CV .
3. Ho . Jonathan . Salimans . Tim . Classifier-Free Diffusion Guidance . 2022 . cs.LG . 2207.12598.
4. Karras . Tero . Laine . Samuli . Aila . Timo . 2020 . A Style-Based Generator Architecture for Generative Adversarial Networks . IEEE Transactions on Pattern Analysis and Machine Intelligence . PP . 12 . 4217–4228 . 1812.04948 . 10.1109/TPAMI.2020.2970919 . 32012000 . 211022860.
5. 1912.04958 . cs.CV . Tero . Karras . Samuli . Laine . Analyzing and Improving the Image Quality of StyleGAN . 23 March 2020 . Aittala . Miika . Hellsten . Janne . Lehtinen . Jaakko . Aila . Timo.
6. Jayasumana . Sadeep . Ramalingam . Srikumar . Veit . Andreas . Glasner . Daniel . Chakrabarti . Ayan . Kumar . Sanjiv . 2024 . Rethinking FID: Towards a Better Evaluation Metric for Image Generation . en . 9307–9315.
7. 2104.08718 . cs.CV . Jack . Hessel . Ari . Holtzman . CLIPScore: A Reference-free Evaluation Metric for Image Captioning . 2021 . Forbes . Maxwell . Ronan Le Bras . Choi . Yejin.
8. Lin . Tsung-Yi . Microsoft COCO: Common Objects in Context . 2015-02-20 . Maire . Michael . Belongie . Serge . Bourdev . Lubomir . Girshick . Ross . Hays . James . Perona . Pietro . Ramanan . Deva . Zitnick . C. Lawrence. cs.CV . 1405.0312 .
9. Fréchet. . M . 1957 . Sur la distance de deux lois de probabilité. ("On the distance between two probability laws") . C. R. Acad. Sci. Paris . 244 . 689–692 . translated abstract: The author indicates an explicit expression of the distance of two probability laws, according to the first definition of Paul Lévy. He also indicates a convenient modification of this definition..
10. Dowson . D. C . Landau . B. V . 1 September 1982 . The Fréchet distance between multivariate normal distributions . Journal of Multivariate Analysis . en . 12 . 3 . 450–455 . 10.1016/0047-259X(82)90077-X . 0047-259X . free.
11. Kilgour . Kevin . Zuluaga . Mauricio . Roblek . Dominik . Sharifi . Matthew . Fréchet Audio Distance: A Reference-Free Metric for Evaluating Music Enhancement Algorithms . Interspeech 2019 . 2019-09-15 . 2350–2354 . 10.21437/Interspeech.2019-2219. 202725406 .
12. Open Review . Unterthiner . Thomas . Steenkiste . Sjoerd van . Kurach . Karol . Marinier . Raphaël . Michalski . Marcin . Gelly . Sylvain . FVD: A new Metric for Video Generation . 2019-03-27 . en.
13. Preuer . Kristina . Renz . Philipp . Unterthiner . Thomas . Hochreiter . Sepp . Klambauer . Günter . Fréchet ChemNet Distance: A Metric for Generative Models for Molecules in Drug Discovery . Journal of Chemical Information and Modeling . 2018-09-24 . 58 . 9 . 1736–1741 . 10.1021/acs.jcim.8b00234. 30118593 . 1803.09518 . 51892387 .
14. Chong . Min Jin . Forsyth . David . 2020-06-15 . Effectively Unbiased FID and Inception Score and where to find them . cs.CV . 1911.07023 .
15. Liu . Shaohui . Wei . Yi . Lu . Jiwen . Zhou . Jie . 2018-07-19 . An Improved Evaluation Framework for Generative Adversarial Networks . cs.CV . 1803.07474 .