Breaking Semantic Artifacts for Generalized AI-generated Image Detection

Thank you for taking the time to read our manuscript and for providing detailed comments! Below are our responses to the comments provided:

Q1: Comparison to (Dogoulis et al. 2023), which has discussed the problem of performance drop in cross-scene settings.

First, we acknowledge that the cross-scene generalization problem is well recognized, and our contribution is not to define the problem but to better solve it. Tables A and B in the PDF attached with the global response show that our method largely surpasses (Dogoulis et al. 2023) by 32.38% in Acc. and 29.65% in AP on open-world evaluation.

See the following for the explanations of their poor performance and the experimental settings. The original setting in (Dogoulis et al. 2023) is less challenging because they only focus on generalization between different concept classes (e.g., objects) but do not involve more diverse contents, represented by the LAION, as in our work. Therefore, their assumption is essentially not sufficient for text-conditional generation. We notice that they originally select the top-10k images of 20k generated images as training sets, while we have 48k generated images. For a fair comparison, we consider two settings: A) top-10k images and B) top-24k images. We followed their official codes and selected ResNet-50 as the backbone. Gaussian blurring and JPEG compression are applied as data augmentation with a probability of 10%.

Q2: Missing discussion and comparison on the influence of resizing the images.

See our response in Q3 and Q4 of Reviewer Zwma. In sum, we add ablation experiments and find that padding is a better choice than image resizing.

Q3: Big performance gaps between Acc. and AP are still observed in Table 5.

As seen in our response in Q3 and Q4 of Reviewer Zwma, the most considerable gaps on specific generators, e.g., SITD, SAN, CRN, IMLE, and WFIR can be solved by replacing the resizing with padding. For example, Table B shows that our method reaches the SOTA on cross-GAN/CNN (Acc. 80.13%, AP 84.97% vs. Acc. 72.58%, AP 81.38% of the second best method, NRP. The ACC-AP gap decreases from 8.8% to 4.84%.

Q4: The method is extremely simple and nothing is methodologically novel. Close performance compared with the SOTA baseline.

Based on our new results in Table B, our method substantially improves the previous SOTA, NPR, from ACC. 75.38% to 85.97% in the open-world evaluation.

We welcome further improvements on top of our work. However, we believe the simplicity of our method does not diminish its novelty. Conceptually, our novelty lies in introducing the concept of "semantic artifacts" and demonstrating that breaking "semantic artifacts" is the key to cross-scene generalization. Based on this finding, we propose a novel design of a patch shuffle-based end-to-end detection framework.

Q5: Sections 3.1 and 3.2 are not part of the methodology in my opinion, they should be placed under a separate motivation section as preliminary/insightful experiments. Section 3.3 although very small is still verbose and gives unnecessary details.

We will further smooth the logic in Sections 3.1 and 3.2, and move unnecessary details to the Appendix.

We agree that Sections 3.1 and 3.2 are not directly about the methodology, but we group them together with Section 3.3 to ensure a smooth transition from the motivation about "semantic artifact" to the method design.

Thank you for your comments! Below are our responses to the comments:

Q1: Idea-level comparison to PatchFor [6].

The key idea-level difference between PatchFor and our method is the use of patch shuffle. This is the core of our method, which ensures that the model completely removes the semantic information as a whole, thereby mitigating overfitting to semantic artifacts. Another technical difference is that we force our model to accept only patch inputs but PatchFor still accepts the full image, which contains rich semantic information.

Q2: Experiments using the images collected from fake news on the Internet.

We have already followed the common practice [4,7,8] for datasets and experimental settings. Notably, these datasets include data from the Internet, such as Deepfake (sourced from YouTube videos) and WFIR (sourced from websites). For the fake news, we believe it is clearly a valuable direction for future work but technically, it is infeasible because the ground truth of the fake news on the Internet is hard to get, making the model training and evaluation not possible.

Thank you for the feedback and comments! We are glad to hear that you believe this is important work and that it meets the standard for publication. Below are our responses to the comments provided:

Q1: Missing frequency visualizations on patch-shuffled images as well as image patches.

Thank you for your suggestions. The frequency visualizations of shuffled images and image patches are provided in Figure A of the PDF attached with our global response.

These visualizations support our original hypothesis. Specifically, based on the visualization of shuffled images, most artifacts are removed during the patch shuffling. For instance, the distinct artifacts between CelebA and LAION or between LDM-CelebA and LDM-LAION are significantly reduced. In addition, the visualizations of image patches reveal an intriguing finding that low-frequency features are weakened but high-frequency features (corresponding to artifacts) are enhanced.

Q2: Not clear what artifacts are in L178-180 and why the visualization in Figure 3 should be more focused on point regions.

We will make it clear that the "artifacts" in L178-180 refer to "generator artifacts", as defined in Section 3.1.

For visualizations in Figure 3, we will clarify that the generator artifacts generally correspond to more universal, point regions, which represent a deeper receptive field, meaning a higher-level feature extraction. So it can be seen that our method successfully directs the model to focus on these regions rather than the concentrated, semantic regions.

Q3: Missing discussion and visualization on the influence of image resizing.

We acknowledge that the use of image resizing is not a particular design. In our additional experiments, we also test image padding and find it to be more effective. See Figure A and Tables A and B in the PDF attached with our global response.

Tables A and B demonstrate the superior results of zero padding compared to resizing, particularly in cross-GAN/CNN generalization. In particular, the performance of our method has been largely boosted on SITD, SAN, CRN, IMEL, and WFIR. It can be attributed to their high image resolution, which introduces variations in artifacts and leads to the loss of low-level features when resizing is applied. Figure A supports this finding by showing that the frequency features of SITD (with image resolution of over 4,000×3,000) images change a lot after resizing.

Note that on ganGAN, resizing is better than padding probably because their images have been resized during the original data collection.

Q4: Missing analysis of why the proposed approaches have considerable gaps with existing approaches on specific generators.

The considerable gaps (on SITD, SAN, CRN, IMEL, and WFIR) have been explained in the above response.

Q5: Missing comparison and discussion about "Towards Discovery and Attribution of Open-world GAN Generated Images".

The suggested paper and our work focus on different tasks. They aim at attribution and discovery of generated images, rather than our (generalized) detection. Therefore, a direct comparison is not possible. Technically, their method relies on out-of-distribution detection and clustering while ours directly destroys artifacts.

Q6: Small corrections: L119 bot -> both L119 synthetics -> synthetic

Thanks. We will fix them.

Thank you for taking the time to read our manuscript and for providing detailed comments! Below are our responses to your comments:

Q1: Missing ablations of patch size on the performance.

We have already reported the ablation results of patch size (and model depth) in Appendix A.1.1, and we will add conclusions in the main text based on them. The superiority of our method holds for all tested patch sizes and model depths.

Specifically, a too-large patch size enlarges the receptive field, potentially exacerbating the overfitting issues, while a too-small patch size destroys certain low-level semantic features, potentially causing underfitting.

Q2: Missing visualizations of real images in Figure 3.

We add such visualizations to the attached PDF in our global response, as illustrated in Figure B, and we will include them in our updated manuscript.

As expected, the CAM visualizations of all the detectors on real images (i.e., with the label "0") show almost no activated regions. This aligns with the high Acc. results of real images in cross-scene settings (see Table 1).

Q3: Missing details about the data selection and design.

We have already detailed the dataset collection in Appendix A.4, and we will summarize them in the main body of our updated manuscript.

Specifically, for DM-generated images, our selected generative models span a long time range, from 2020 to now. They fall into 3 categories: unconditional, class-conditional, and text-conditional. For real images, the training data of each generative model are used to ensure the same distribution to their generated images.

Q4: Missing citations of [1, 2], which discuss the use of patches to mitigate the influence of semantic features.

Thank you for recommending these two papers, which can be used to better motivate our use of patches. We will cite them in our updated manuscript.

Q5: Data pairing as a potential baseline.

This is indeed a good point! A similar idea has been adopted by very recent methods, e.g., DIRE [54], which models the error between an input image and its reconstructed counterpart by a pre-trained diffusion model. We have validated that our method outperforms DIRE by 25.07% in Acc. and 24.55% in AP on open-world evaluation (see Table 3).

We believe data pairing has the following limitations:

• It targets specific semantic objects, e.g., church, bedroom, human face, making it feasible only for unconditional or class-conditional generated images. However, for text-conditional generated images, which contain complicated semantics, paired data are hard to define and collect.
• Although paired data can remove the semantic artifacts, the model may still learn the domain-specific information. Then, when it comes to different domains during testing, the performance decreases. For example, a generated human face normally contains most artifacts in its domain-specific regions, e.g., the lineament and hair. This assumption is supported by our results in Table 4, where DIRE (trained on CelebA) degrades from 83.25% Acc. on CelebA-LDM to 50.45% Acc. on LAION-LDM.

In contrast, our method is both tailored to specific semantics and indiscriminately destroys the semantic artifacts.