Manifold Induced Biases for Zero-shot and Few-shot Detection of Generated Images

We appreciate the recognition of the sound theory and robust performance of our approach. Below we answer the questions raised.

Q1: “..it is unclear why this method also performs well with models where score functions are not inherently applicable..”

A1: We appreciate this question, which addresses a fundamental challenge in the field of generative image detection: Mitigating the biases that arise when a method is exposed to specific generative techniques, which could impair its generalization to new, unseen techniques. Although such generalization is a notable strength of our approach, in the Limitations section we acknowledge that we do not have an extensive theory regarding the generalizability to unseen generative techniques.

One possible explanation lies in the shared characteristics of generative model manifolds due to similarity in training data. There is limited availability of large-scale datasets [1,2], and many generative models are trained on similar datasets , which may lead to overlapping or closely related learned probability manifolds. This is despite the differences in architecture, size, and configuration [3]. Consequently, our method, which is exposed to one model’s manifold, may effectively use it as a proxy for detecting images generated by other generative models. This hypothesis aligns with observations in the literature that models trained on comparable datasets exhibit commonalities [4,5].

An interesting hypothetical challenge in the field may arise with the introduction of an entirely novel dataset—one that no generative model has been trained on to date, followed by training of new generative models. In such a scenario, it is uncertain whether existing methods would maintain their performance on the resulting newly generated images, as their success may be tied to biases in currently available datasets. However, we believe that our method can adapt effectively, and that substituting the SD1.4 model with a diffusion model trained on the new dataset is likely to yield strong performance.

To better clarify this in the manuscript, we expanded the discussion in the Limitations section to better outline the shared data perspective and include the references added here.

[1] Villalobos , P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., & Hobbhahn, M. Position: Will we run out of data? Limits of LLM scaling based on human-generated data. In Forty-first International Conference on Machine Learning.‏

[2] https://www.bloomberg.com/news/articles/2024-11-13/openai-google-and-anthropic-are-struggling-to-build-more-advanced-ai?embedded-checkout=true

[3] Nalisnick, E., Matsukawa, A., Teh, Y. W., Gorur, D., & Lakshminarayanan, B. Do Deep Generative Models Know What They Don't Know?. In International Conference on Learning Representations.

[4] Kornblith, S., Norouzi, M., Lee, H., & Hinton, G. (2019, May). Similarity of neural network representations revisited. In International conference on machine learning (pp. 3519-3529). PMLR.

[5] Nguyen, T., Raghu, M., & Kornblith, S. (2020). Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. arXiv preprint arXiv:2010.15327.

Q2: Citation issues on page 7

A2: In the revised version, we have corrected all citation formatting and ensured that footnotes and page numbers are accurately referenced.

Q3: " The authors are encouraged to clarify the need for a few-shot setting and explain why zero-shot alone would not suffice as a more compelling and realistic approach."

A3: We agree that zero-shot scenarios often reflect real-world situations, and our solution is fully suitable for such cases.

Nonetheless, in some scenarios, managing a small amount of generated data can be justified if it leads to significant performance improvements - making few-shot methods appropriate. Our research demonstrates that integrating our method with a SOTA few-shot technique, Cozzolino et al. (2024), yields a 4–7% improvement in detection performance without violating the few-shot setting.

For users willing to invest in data maintenance, our method provides a valuable enhancement to few-shot frameworks. It serves as an easy-to-integrate plugin, offering a flexible trade-off between data availability and performance improvement.

Even though few-shot methods for generated image detection offer practical potential, balancing a trade-off between maintenance and performance, this domain is still largely under-explored (as are zero-shot methods). Our work contributes a practical approach to both regimes, excelling in zero-shot scenarios and enhancing few-shot performance.

We have clarified the motivation for introducing the few-shot evaluation in the revised manuscript in Section 5.2, under Mixture of Experts with Few-shot Approaches.

Thank you for the thoughtful feedback and for the careful attention given to our methodology and mathematical details. Below we provide our answers.

Q1: Improve the implementation details in Section 4.3

A1: Thank you for raising this issue. To improve the readability of Section 4.3 and the ease of reproducibility, the calculation pipeline of $C(x_0)$ was broken down into three clear steps, referencing Fig. 1 to aid with a high-level illustration. Additionally, the CLIP mappings have been restructured into a two-stage process and are now explained in a less formal, more intuitive manner, reducing the overly mathematical approach that previously hindered clarity. We also separated best practices from insights.

Please do not hesitate to raise further questions regarding unclear points or any additional details that may have been overlooked.

Q2: Typos

A2: We have fixed the typos and corrected equation presentations - thank you for finding these.

Q3: Will incorporating to the method other diffusion models that are more advanced than SD1.4 improve its performance?

A3: In Sec. 5.2 under Sensitivity and Ablation Analysis we have readily evaluated our method using two advanced diffusion models: SD v2 Base and Kandinsky 2.1, where the latter was released in September 2023 - the same month as SDXL. We acknowledge this analysis may have been overlooked as it was only mentioned in the text. In the revised manuscript, we have highlighted this experiment by adding a summary table of the results – see Table 2.

Although SD v2 Base and Kandinsky 2.1 models differ in size and generation techniques, our experiments demonstrated consistent results, with a minor AUC decrease of less than 1%. This consistency highlights the robustness of our method across different diffusion models. With that said, it also demonstrates that more advanced models do not necessarily improve the performance.

If further details or additional experiments on this matter would be helpful, please let us know and we would be happy to provide them.

We greatly appreciate the reviewers' recognition of our work and the constructive feedback. This prompted us to conduct additional experiments on robustness (Table 2, Fig. 10) and computational efficiency (Appendix Sec. G.3), strengthening the rigor and validation of our approach. Below we answer the questions in detail:

A1: Thank you for pointing this out. We conducted additional experiments to analyze our method inference time and memory requirements. Using a single A100 GPU, we observed a runtime of 2.1 seconds per sample upon fully parallelized processing of the perturbations. For comparison, our primary competitor, AEROBLADE, requires 5.4 seconds per sample on the same A100 GPU, making our method significantly more computationally efficient in terms of runtime. The additional memory required by our method is negligible, as it applies the diffusion model during inference and records only a single scalar value per sample as the final result. This analysis has been included in the Appendix (see Section G.3) for further reference.

A2: Thank you for highlighting the importance of hyperparameter robustness. We provide sensitivity analysis for the perturbation no. and the spherical noises level hyperparameters in Section 5.2 ("Sensitivity and Ablation Analysis") with further details in Appendix Section G.2.

Regarding the perturbation no., our experiments indicate that increasing the no. of perturbations $s$ improves detection performance. Specifically, testing $s$=4, 8, 16, 32, and 64 resulted in average AUC scores of 0.828, 0.829, 0.830, 0.833, and 0.835, respectively. To address the reviewer’s suggestion, we also evaluated the robustness to perturbation no. across different models. These experiments revealed only minimal variations in performance (0.1–0.2%) with increased $s$ - see Fig. 10 in Appendix Section G.2.

We also evaluate our method under varied spherical noise levels. In our experiment, increasing the radii by a factor of 10 resulted in a 1.5% performance decrease, while increasing by a factor of 100 led to a 2% decrease.

Please note that for improved clarity and presentation, we have added a table in the main manuscript that summarizes the results of this experiment (see Table 2).

These results show that we remain SOTA even under changes in both hyper-parameters. As guidance for noise strength search we have used $\sqrt{d}$ as a point of reference to the sphere's radius, because of its theoretical relation to $\mathcal{N}(0,I)$ in $d$ dimensions (where $d$ is the denoised signal dimension). We also propose to test with higher $s$ and smaller perturbations, since the trend of our tests indicate that these may obtain improved performance.

A3: Thank you for this valuable feedback. Indeed, evaluating our method on various image post-processing options would provide valuable information on the robustness of our method to real-world possibilities. To address this, we follow [1, 2] and test Gaussian blur post-processing.

We conducted experiments using Gaussian blur. Specifically, we applied OpenCV’s Gaussian blurring functionality with two kernel sizes (and the default associated variance): medium blur (Kernel Size = 3) and high blur (Kernel Size = 7). Under medium blur conditions, our method’s accuracy decreased by 1.2%, while under high blur conditions, accuracy reductions were 6.2%.

This analysis has been included in the main manuscript in Section 5.2 ‘Sensitivity and Ablation Analysis’ part and in the Appendix (see Section G.2) for further reference.

[1] Ricker, J., Lukovnikov, D., & Fischer, A. (2024). AEROBLADE: Training-Free Detection of Latent Diffusion Images Using Autoencoder Reconstruction Error. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9130-9140).‏

[2] He, Z., Chen, P. Y., & Ho, T. Y. (2024). RIGID: A Training-free and Model-Agnostic Framework for Robust AI-Generated Image Detection. arXiv preprint arXiv:2405.20112.‏

Thank you for your constructive feedback, which has been instrumental in improving our work. In response, we empirically demonstrate our key assumptions (Figs. 3.a, 4.c and 7) and test the reliability of our approximations (Fig 3.b-d) through analyzeable and illustrative cases.

A1: Thank you for raising the point on “region of local maxima assumption”. To this end we conducted an experiment with a 2D Gaussian Mixture Model (GMM) dataset - see Appendix B for details. This setting allows for tractable Probability Density Function analysis and was inspired by [1] which also used 2D GMMs to gain insights into diffusion models. As shown in Fig. 3.a, reverse diffusion processes consistently terminate near local maxima, supporting our assumption. This 2D experiment offers intuitive, tractable evidence for similar behaviors in high-dimensional models. Further statistics at larger scale support this in Fig.7 in the Appendix.

Furthermore, this assumption aligns with prior observations: Fig. 2 in [2] illustrates a similar assumption for text generation, albeit relying on LLM’s explicit probability modeling. Fig. 2 in [1] shows how a diffusion model learns basins of local maxima, which "overtake" a non-maxima basin present in the true probability distribution. Recently, [3] associated suboptimally trained diffusion models with "bumpy" probability manifolds.

Figs. 3.a, 7 support our specific perspective of the local maxima property. Please let us know of additional tests that may further substantiate it.

A2 : Thank you for the suggestion to provide an error analysis of the curvature approximations. To this end, we conducted an experiment using an analytic 2D function with complex yet “nice” (differentiable) topography, enabling true quantities for validation.

The curvature $\kappa$ (Eq. 8) around a data point is approximated through discrete sampling of its neighborhood boundary. Though grounded in Gauss Divergence Thm., we recognize that the finite sample size warrants error analysis. The findings, presented in Fig. 3 (b-d), demonstrate the following:

1. Expected Behaviour: The Local maxima receives higher true $\kappa$ than the saddle point (Fig. 3.b).

2. Robustness to Low-Sample Approximations: Even at low sample sizes with large error bars, maxima and saddle points are separable by a threshold (Fig. 3.c). This separability is the focus of our paper.

3. Consistent Estimator: The $\kappa$ estimates’ mean is close to the true value across sample sizes (Fig. 3.c), while their std decays exponentially (Fig. 3.d).

This experiment provides essential reliability verification of our theoretical framework. If any important statistics can be added - please inform us. Details are in Appendix A.

A3 : We would like to kindly draw your attention to the fact that the bias mentioned in line 122 refers to prior methods that train detectors on generated images. Such training introduces significant bias to the generation techniques encountered during training as documented in Fig. 2 of [4] and Fig. 2 of [5]. In contrast, our zero-shot method is training-free leveraging a pre-trained SD1.4. Importantly, SD1.4 was pre-trained on real images (without generated images).

We acknowledge the bias potential inherent to using a specific model (SD1.4). To test our method in this regard, we evaluated our approach on over 100K generated and 100K real images across 20 generation techniques – some introduced after SD1.4. Our approach shows strong generalization to unseen generation techniques (Fig. 5, Table 1), providing thorough evidence that it is not confined to the biases of SD1.4.

A4 : Another important concept that requires an illustration is the concentration of measure, which describes how high-dimensional Gaussian samples concentrate around a sphere - a phenomenon critical to our derivations. To this end we added an additional 2D illustration in Fig 4.c as follows:

1. For a $d$-dimensional $\epsilon\sim\mathcal{N}(0,I)$ we obtained the $\chi$-distributed $E\|\epsilon\|$, $\mathrm{Var}(\|\epsilon\|)$ (norm’s mean and variance).

2. We visualized corresponding 2D samples scaled to have these mean and variance of their norm, effectively simulating the phenomenon in 2D.

As $d$ increases, the radius becomes larger, and the variance converges, empirically illustrating the expected “thin shell” spherical distribution.

References

[1] Song & Ermon, NeurIPS 2019, "Generative modeling by estimating gradients of the data distribution"

[2] Mitchell, et al., ICML 2023, "Detectgpt: Zero-shot machine-generated text detection using probability curvature"

[3] Zahra Kadkhodaie et. al, ICLR 2024, “Generalization in diffusion models arises from geometry-adaptive harmonic representations”

[4] Epstein et. al, ICCV 2023 “Online detection of ai-generated images”

[5] Ojha et. al., CVPR 2023, “Towards universal fake image detectors that generalize across generative models”

Thanks for the detailed clarification. I have checked the revised materials. As my concerns are addressed, I will raise my rating accordingly.