Fréchet Wavelet Distance: A Domain-Agnostic Metric for Image Generation

We address your questions as follows:

_Q: Metrics such as IS (Inception Score), KID (Kernel Inception Distance), $FID_\infty$, $IS_\infty$, and Clean FID should also be considered

We appreciate the reviewer for highlighting additional metrics. Our proposed metric FWD is designed to address domain bias, a critical issue in generative image evaluation as identified by [Kynkäänniemi et al., 2023]. As for the other metrics, we would like to point to the related work section of our paper. In brief, IS was widely used but found to be sensitive to multiple perturbations, does not consider real data distribution [Heusel et al., 2017], and is sensitive to various training runs [Barratt & Sharma, 2018]. Our metric FWD, as well as FID and FD-DINOv2, are robust in these aspects, as illustrated in Figure 7. Secondly, $FID_\infty$ and $IS_\infty$ proposed by [Chong et al., 2019] are built to extrapolate FID and eliminate sample size-related bias. Clean FID [Parmar et al., 2020] studies compression-related artifacts. [Kynkäänniemi et al., 2023] demonstrated that KID and FID are susceptible to domain bias. Consequently, domain bias remains unaddressed, which is central to our metric.

To provide an analysis, we report our results in the table below. The result demonstrates susceptibility to domain bias. Overall, this highlights the importance of FWD, which is specifically designed to address the domain bias problem in generative models. We believe this extended comparison addresses the reviewer's concern and reinforces the significance of FWD as a robust, fast and computationally efficient metric for tackling domain bias in evaluating generative models.

Q:lack of human evaluation

We refer to the overall rebuttal response above that addresses this concern.

Q:The underlying mechanism is not entirely clear

Interestingly, the Wavelet Packet Transform (WPT) enhances transparency in the following way. The WPT splits the input image into various frequency bands (low -> high ) based on the transformation level, as illustrated in Figure 2. We then compute Frechet Distance (FD) across individual frequency bands, and this individual FD score (Figure 6) provides valuable insights into the model's performance across different frequency bands. For instance, Figure 6 demonstrates that DDGAN achieves a better Frechet Wavelet Distance (FWD) score than StyleGAN2 because DDGAN can generate a better frequency response.

Q: The dataset selection is limited.

The proposed metric is specifically designed to detect domain biases, and our dataset choices CelebAHQ, FFHQ, DNDD and Sentinel were intentionally selected as they are out-of-domain for ImageNet pretrained metrics like FID. This selection allows us to demonstrate the domain bias in existing metrics effectively. For larger datasets such as LAION and LVD-142M, determining the extent of their in-domain coverage is challenging. For instance, in the case of DINOv2 pre-trained on LVD-142M, we hypothesize that agriculture images are not part of LVD-142M's training set. As a result, FD-DINOv2 produces a biased estimate in this case.

Moreover, no dataset is truly universal, and out-of-domain scenarios will always exist. To identify these, we carefully designed FWD using the WPT, which is independent of the pretraining dataset, delivers domain bias-free estimates, is highly reproducible (refer to sdD9 response), tested (supplementary zip contains code), and is cheap to compute.

References

[Kynkäänniemi et al, 2023] Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. ICLR 2023.

[Barratt & Sharma, 2018] Shane T. Barratt and Rishi Sharma. A note on the inception score. _ICML 2018 workshop.

[Heusel et al, 2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Neurips 2017,

[Chong et al, 2020] Chong, Min Jin and Forsyth, David. Effectively Unbiased FID and Inception Score and where to find them. CVPR 2020.

[Parmar et al, 2020] Parmar, Gaurav and Zhang, Richard and Zhu, Jun-Yan. On Aliased Resizing and Surprising Subtleties in GAN Evaluation. CVPR 2020.

For clarity, we provide the complete table including FID, FD-DINOv2 and FWD here

Due to space constraints, we cannot include the full table in the rebuttal above.

Thank you for your questions, we will address these below:

make the new FWD metric more convincing, a human subject experiment is necessary

We refer to the overall rebuttal response above that addresses this concern.

Many image quality metrics have been proposed for generative model evaluation, but to my knowledge, FID is still the mostly adopted one. I want to hear the authors opinion on this problem.

We thank the reviewer for this interesting observation. At the time Frechet Inception Distance (FID) was introduced [Heusel et al, 2017], the prevalent metric for evaluating images was Inception Score (IS). However, FID demonstrated that IS is susceptible to various types of noise and doesn't account for the original data distribution. FID emerged as an alternative to IS during the rise in popularity of GANs, and its utility as a robust metric led to its widespread adoption. While alternatives such as KID, FID_Infinity, and CleanFID have been proposed, FID still remains the metric for benchmarking generative models against prior literature.

This fact forces authors to work with FID for comparisons. Furthermore, IS, KID, FID_Infinity, and CleanFID all share the ImageNet-pre-trained InceptionV3 backbone with FID. With the shared backbone comes a shared domain bias problem. This paper addresses this problem.

This is why our study includes evaluations across well-established GAN models such as StyleGAN2, StyleSwin, BIGGAN, and Proj. FastGAN and recent diffusion architectures like EDM, Improved Diffusion, DDPM and DDIM. Despite its effectiveness, FID has notable limitations, such as susceptibility to domain bias [Kynkäänniemi et al, 2023] and compression algorithms [Parmar et al, 2020]. To address these shortcomings, we propose FWD as an additional metric designed to detect domain bias, which existing metrics fail to capture [Table 1].

Will evaluating on SOTA generative models like stable diffusion series help the advocation of a new image quality metric?

We are working on this point, we will report back once results become available.

References

[Heusel et al, 2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017a.

[Kynkäänniemi et al, 2023] Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. ICLR 2023, Kigali, Rwanda, 2023.

[Parmar et al, 2020] Parmar, Gaurav and Zhang, Richard and Zhu, Jun-Yan. On Aliased Resizing and Surprising Subtleties in GAN Evaluation. Computer Vision and Pattern Recognition. 2020.

We want to address your questions as follows:

Q:Limited Comparison with Alternative Frequency-Based Metrics

We appreciate the reviewer's interest in alternative frequency-based metrics. Among the current metrics, we are aware of Sliced Wasserstein Distance (SWD) and Multiscale Structural Similarity Index Measure (MS-SSIM) as reported by [Karras et al, 2018]. However, each of these metrics presents unique challenges

1. SSIM: SSIM is an effectice pair-wise metric suitable for cases like Super resolution. It isn't relevant when the images are unconditionally generated as the corresponding reference images aren't available.
2. SWD: While SWD can detect domain bias, it suffers from significant reproducibiltiy issues, due to the projection onto an independent uniform random basis [Nguyen et al, 2023]. The following table compares the performance of SWD and our proposed metric FWD across five independent runs. SWD demonstrate substantially large standard deviation, whereas FWD remains consistent regardless of number of runs.

Given these limitations, and known bias problem with FID and FD-DINOv2, we believe it is crucial to develop a metric which can solve the domain bias reliably. Furthermore, our careful adoption of Wavelet Packet Transform (WPT) not only allows for frequency-band interpretation but also reduces spatial dimensions, simplifying the covariance matrix computation in Frechet Distance.

Q: Potential Alignment with Human Evaluation

We refer to the overall rebuttal response above that addresses this concern.

References

[Nguyen et al, 2023] Nguyen, Khai, Tongzheng Ren, and Nhat Ho. "Markovian sliced Wasserstein distances: Beyond independent projections." Advances in Neural Information Processing Systems 36 (2023).

[Karras et al, 2018] Progressive Growing of GANs for Improved Quality, Stability, and Variation, ICLR 2018.

We sincerely thank the reviewers for their feedback and active participation in the discussion. We have updated the paper to include results from the user study and additional metrics in the supplementary material, adhering to the page limit regulations set by ICLR. Regarding the stable diffusion experiments, for reviewer whKw, we so far provide results for Stable Diffusion 2 and Stable Diffusion XL. As the additional experiments requested by sdD9 involve generating images using other stable diffusion models, which require more time to complete, we plan to include these results in the camera-ready version and will notify you once the results become are available. We hope that, based on the discussion so far, we could convince you that our paper will effectively address those open points. Thank you for your time and consideration.