Are Images Indistinguishable to Humans Also Indistinguishable to Classifiers?

My main concern with the paper is that while the paper includes a substantial amount of data and findings, it lacks a cohesive narrative that ties the results together and clarifies their implications.

1. For example, the paper demonstrates differences between FID and classifier results (sections 5.2 and 6) without in-depth discussions on the reasons behind these discrepancies. Section 7.1 shows some analysis on what the classifiers focus on, it would be helpful to perform similar analysis for the FID metrics, and pointing out the differences in what aspects of the images the two methods focus on.
2. Similarly, observations like the classifier’s poor performance with CIFAR data are presented without verification of potential causes (section 5.3). Testing with CIFAR10-pretrained self-supervised classifiers would strengthen these claims. For example, are diffusion models really closing the gap between real and generated images on low-resolution datasets?
3. Additionally, the purpose of using self-supervised models in classification (section 5.3) is unclear, and further context would help clarify their role.

Other concerns:

1. The analysis of models within the same family but with different capacities is limited to only one model (EDM2-XS and EDM2-XXL), which weakens support for this conclusion. Including additional models of varying capacities would provide more evidence.
2. The user study involves only 19 participants, which could introduce noise or bias. More information on participant agreement rates/variance of the results would enhance the reliability of these results.

1. The authors empirically discover interesting inconsistencies between classifier performance, human evaluations, and FID metrics but lack in-depth analysis of the underlying reasons. This results in relatively weak theoretical contribution.
2. While classifiers are introduced as a new evaluation method, the paper lacks a detailed discussion of where and why this approach is meaningful. The significance of using classifiers as a metric remains unclear.

• This work has a lack of providing any theoretical analysis or theoretical intuitions to explain why the classifier, even self-supervised classifier can easily distinguish the generated data from real data, when human and FID score cannot see the differences.

• The authors may provide more details of FID metric and the proposed classifier-metric. They may explain the differences of the distances between the generated distribution and the real distribution provided by FID score and the distance provided by classifiers. Analyzing the differences may help us to understand the contradictions of FID score and classifiers for measuring the generated distributions.

1. One of my main concerns is that many of its findings seem to echo results from previous works, such as those cited in the related work section. Prior research has demonstrated that neural network classifiers can effectively distinguish real from generated images, which somewhat diminishes the unique contribution of this study. Without extending or innovating upon these prior insights, the paper's impact remains limited in originality.

2. The paper frequently refers to its approach as "distribution classification," implying a theoretical or practical connection to statistical concepts of distribution. However, the methodology appears identical to regular classification tasks focused on distinguishing real and generated images. This terminology may be misleading, as it does not introduce any specific statistical treatment or distributional modeling. Greater clarity around this term and why it differs from conventional classifier settings would be beneficial for readers.

3. The methodology mainly employs existing classifiers and conventional metrics, such as FID and classification accuracy, to evaluate generative models, leading to limited methodological novelty. The reliance on standard architectures (e.g., ResNet-50, ConvNeXt, ViT-S) and classification accuracy as a primary metric for assessing generative quality does not offer fresh insights or approaches to the problem. Coupled with the limited novelty in findings, this contributes to the perception that the paper lacks substantial innovation.

4. While the classifier-based evaluation offers diagnostic insights, the paper does not provide actionable improvements or enhancements for current generative models based on those insights.

W1. Prior work has shown the neural network-based classifiers are sensitive to high-frequency features/noises. Conducting a more in-depth analysis of the specific features that classifiers focus on could strengthen the paper, especially from an explainability perspective. For instance, the observation that samples from the same model family are harder to distinguish compared to samples from different model families raises the question of whether each model family has a unique “watermark” embedded in their generated images.

W2. Some related work should be discussed and compared, e.g., [C1] showed FID doesn't correlate well with human judgements, and [C2] on different metrics and low-level and high-level features.

[C1] Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models, Stein et al., NeurIPS 2023

[C2] Dreamsim: Learning new dimensions of human visual similarity using synthetic data, Stephanie et al., NeurIPS 2023