PiD: Generalized AI-Generated Images Detection with Pixelwise Decomposition Residuals

We sincerely thank Reviewer 4ooD for the thoughtful comments. The responses to the questions are as follows.

Q1: Why do these residuals inherently capture the artifacts?

A: Thanks.

• Generally, the residual for an image input $x$ has the form $R(x) = x - \Phi(x)$ as described in the paper. Ideally, $\Phi(x)$ represents the information that contributes most to the visual quality.
• The training of generative models mainly focuses on content consistency, while the noise information is not well-modeled (as shown in Eq. 3). This information does not affect the overall visual quality of images, but can also reflect the difference between real and fake images.
• Differences in high-frequency residual distribution are observed in [1]. While the distribution of other residuals cannot be well visualized, the test results demonstrate that residuals like DIRE or the proposed PiD also exhibit discrimination capability in AIGC detection.

Q2: Other residual processing methods.

A: Some methods in previous work can also be classified as residual-based methods like high-frequency components (DCT/DWT/FFT) or reconstruction error (diffusion models). An encoder-decoder structure exists in these methods. However, learnable structures may be avoided to alleviate the bias in the residual signals. A simple operation that filters the content information shows advantages in generalization, as the proposed method.

Q3: Results on facial synthesis datasets.

A: Thanks. The generalization of different source images is important. Some test sets in experiments are facial synthesis datasets, in which both real and fake images are faces, like AttGAN, STGAN, and Deepfakes. Our method can generalize well on these facial synthesis datasets at testing. To further explore the capability of the proposed method, we also test our method on high-quality facial synthesis data generated from HeyGEN (VFHQ as the real dataset), the accuracy is 86.95% (much higher than the baseline 62.88%), which shows the potential in broader application.

[1] Frank J, Eisenhofer T, Schönherr L, et al. Leveraging frequency analysis for deep fake image recognition[C]//International conference on machine learning. PMLR, 2020: 3247-3258.

We sincerely thank Reviewer 45ju for the detailed comments. The responses to the questions are as follows.

Q1: Computational efficiency.

A: Thanks. We compare the computation cost of our method and previous methods. The inference time is only slightly slower than the baseline model, while it is significantly faster than other residual-based methods or pretraining-based methods like UnivFD.

Q2: Questions on the training paradigm.

A: Thanks. We emphasize the generalization capability of methods rather than the training paradigm. Therefore, to have a fair comparison with previous work, we follow the same setting commonly used in AIGC detection (training on seen models and testing on both seen and unseen models). Improvement has been demonstrated in the experiments and meet our claim.

Q3: Questions on end-to-end training and networks.

A: Thanks. We clarify the training process as follows.

• 1. Without guidance, networks are not guaranteed to focus on the residual information to classify the data in the training set. The decomposition filters out non-residual information and forces the model to leverage the residual information in classification.
• 2. Residual input is still in an image form, therefore using a convolutional network to capture the patterns is a natural choice as in previous work like FreqNet/DIRE/NPR. The results fairly reflect the capability of different inputs.

Q4: Module settings (image channels, transformation matrix, and quantization).

A: Thanks. We explain the questions on the experiment settings as follows.

• 1. For grayscale or monochrome images, we expand the channels from 1 to 3 to achieve consistent channel numbers during training.
• 2. A full-rank $3\times3$ transform matrix is simple and invertible. It can map an image forward to a different color space and back to the RGB space to compute the residual information. This is commonly used in color transformation as a basic setting.
• 3. Rounding and truncation (floor) are two main basic and standard quantization operations in signal processing and computation [1]. Using standard quantization functions is computationally efficient and demonstrates the effectiveness of the framework. We would like to clarify these points in the manuscript.

Q5: Comparison with camera-specific artifacts.

A: NoisePrint++ is a kind of image representation for image anti-spoofing extracted with networks, rather than an explicit decomposition of image information. We compare the NoisePrint++ as input with our method using the same backbone networks. The accuracy (%) results are as follows. On some test sets, it shows improvement on some test sets but the overall generalization is still limited in AIGC detection. The leverage of learnable image representation in AIGC detection requires further study.

Q6: Questions on the benchmarking.

A: Thanks for the suggestion.

• 1. We intend to compare our method with the best-reported results under the same test setting. Therefore, we cite the higher UnivFD result of 88.8 here. However, the results in DRCT are convincing, and we have reproduced the results of UnivFD with an accuracy of 80.7. This result can be included in the table for future reference.
• 2. The results of UnivFD are not deliberately chosen in Tables 1 and 2. Since we follow the SOTA results from C2P-CLIP in Tables 1 and 2, some of the reported results of early methods like UnivFD are cited as slightly different from the original paper. We would like to cite the best-reported results under the same setting and check them accordingly.

Q7: Suggestions on the reference, writing, and presentation.

A: We have followed your suggestion and modified the manuscript accordingly. The citation is added and discussed in the related works.

Q8: Results on Self-Synthesis.

A: Since the results using round are slightly better on GenImage and UniFakeDetect, we use round as the quantization when testing on Self-Synthesis. The results (Acc/AP) of the floor are as follows, and the performance is still similar.

[1] Quantization. IEEE transactions on information theory.

We sincerely thank Reviewer uAyf for the constructive comments, and the responses are as follows.

Q1: Include real image datasets of different distributions.

A: Thanks. The real image distribution is an important factor during testing. The test datasets UniversalFakeDetect and Self-Synthesis used in the experiments are composed of multi-source test sets. The test real images are from COCO, ImageNet, LAION, LSUN, CelebA, and FF++, and the training set used from ForenSynths is single-source (only 4-class LSUN images). Therefore, the test setting has met the requirement of diversity. We will make it clearer as in previous work.

Q2: More qualitative results.

A: Thanks for the suggestion. We use GradCAM and residual images to illustrate that the model trained with the residual signal attends to more AIGC artifacts correctly than the baseline model. The activation of the baseline model falsely appears on the real images with the RGB input. We would like to provide the visualization of different residuals (frequency or reconstruction) on different generative models to compare them in the manuscript or supplementary material.

Q3: Performance on GLIDE and BigGAN.

A: Thanks. Although the training method is crucial to the generalization performance, the training data distribution is still important. There might be two main reasons why using GLIDE and BigGAN as the training set harms the generalization.

• First, the data of GLIDE and BigGAN in GenImage has a smaller image size than other classes. The resolution of BigGAN is the smallest ($128\times 128$) while the resolution of real images is closer to $512\times 512$. Directly training on them may cause a bias corresponding to the image size.
• Second, the image quality of GLIDE and BigGAN is visually worse than other generative models in GenImage. Images are blurred in GLIDE, and the object is unclear in BigGAN subsets. This causes a large gap in generative artifacts with other models.

Though achieving great generalization performance is difficult in these two training sets, our method largely improves the overall performance and shows an advantage under hard settings.

We sincerely thank Reviewer jvUK for the acknowledgement and constructive feedback. The response is as follows.

Q1: Computation efficiency.

A: Thanks for the advice. To prove the computation efficiency of the proposed method, we compute the inference time (single forward pass) and the size of detector networks for different methods as follows. The transformation introduced in our method only adds a little overhead to the baseline cost. Compared with previous methods using reconstruction and frequency operation, the transformation is efficient. The model is also lightweight compared with methods relying on VLM like UnivFD or C2P-CLIP.

Q2: Dataset used in Table 6.

A: Thanks. Datasets used in Table 6 are subsets of GenImage, like in Table 5. We will clarify it in the manuscript.

Q3: Difference between Image Compression Residuals and Image Reconstruction Residuals.

A: Thanks. The difference between these two concepts is as follows.

• Image Reconstruction Residuals used in the paper specifically describe the reconstruction error of generative models used in previous works like DIRE.
• The concept of Image Compression Residuals is more general in this paper. The compression operation requires an encoder-decoder structure that is not necessarily a neural network (e.g., JPEG compression or PiD). The bias of the generative model may not appear in Image Compression Residuals. We will clarify the concepts in the manuscript.

Q4: Analysis on the transformation.

A: Thanks. We find that the generative model mainly focuses on the content consistency and hypothesize that the overlooked noise information is discriminative in AIGC detection. To verify the hypothesis, we propose the PiD to extract the noise residual during detection.

• Noise contributes little to the training with original input. Suppose the image input is $x = u + \epsilon$ (simplified as vector), considering the simple first linear transformation $f(x) = Wx$ in the network with the loss $L$. The gradient of parameter $W$ can be decomposed into two parts $\frac{\partial L}{\partial f}\frac{\partial f}{\partial W} = \frac{\partial L}{\partial f}x^T = \frac{\partial L}{\partial f}u^T+\frac{\partial L}{\partial f}\epsilon^T$. Given $|u| \gg|\epsilon|$, the gradient is dominated by the main component $u$.

• PiD filters out the main component while maintaining the noise information. To verify the effectiveness of the noise part, we subtract the main component $u$ with the transformation and quantization techniques (noted as $T(\cdot)$). The residual $R(x) = x - T(x) = u + \epsilon - T(u)$. Given that $T(u) = u + \epsilon'$, $R(x) = \epsilon - \epsilon'$ is a proxy representation of image noise, where $\epsilon'$ is controlled by the transformation matrix. Taking $R(x)$ as input, the contribution of the noise part can be verified by the experiments.

Since the YUV transform is integrated in the JPEG algorithm, the quantization noise seems a common noise source for real images. And the generative model may also have special noise patterns mixed with other noise sources. By applying the transform and quantization, the main components of images are filtered out, and the noise patterns can be learned.

We sincerely thank Reviewer 6H4b for the thoughtful and constructive feedback. The response is as follows.

Q1: Performance on related datasets.

A: Thanks for the advice. We conducted the perturbation experiments on the GenImage dataset (SDv1.4 as an in-domain test set and other models as out-of-domain test sets). After the perturbation of JPEG and Gaussian blur (GB), the results are as follows. The performance drop (AP) is similar to the baseline model with normal RGB input, while the overall results remain higher than the baseline model. We will test the model on the source identification dataset VISION.

Q2: Comparison between FFT and DCT.

A: Thanks. Although the DCT part is not the main component in our method, we further ablate the performance with DCT/FFT low/high-frequency components as input on GenImage. The accuracy cannot fully reflect the importance of frequency bands in AIGC detection. From the AP results, high-frequency bands (both DCT and FFT) perform better than low-frequency bands, which meets the results from previous works like FreqNet. However, frequency-based residual inputs seem sensitive to the selection of bands and cannot generalize well on some test sets. Furthermore, JPEG quantizes both high-frequency and low-frequency components in compression to different degrees, and the residual used in the paper is not pure low or high frequencies.

Q3: Comparison between computation costs.

A: We compare the computation cost as follows. We only count the parameters and flops of the detector network for all methods with the fvcore package. The inference time for a single forward pass includes the time of processing the input. The reconstruction-based method, like DIRE, takes the longest due to the extra generation process. Frequency-based operations like FFT in FreqNet and large pre-trained CLIP models used in UnivFD and C2P-CLIP also hinder the reference speed to different degrees. Our method only adds a little overhead compared with the RGB baseline and is efficient in computation.

Q4: Baseline explaination.

A: The baseline model is a simple ResNet model used in NPR (smaller than ResNet18). Our method is also based on the same model. We will make it clearer in the new version.

Q5: Robustness to adversarial attacks

A: The white-box attack is hard to defend without a specially designed strategy. However, the success defense rate has improved significantly compared with the baseline model. The accuracy of the baseline model drops to 0 under a PGD attack and to 0.43 under an FGSM attack. Our model has an accuracy of 0.51 and 0.46 under PGD and FGSM attacks. The robustness has been largely improved under the PGD attack. A randomly selected model ensemble or training with adversarial samples may further alleviate the risk of attacks.

Q6: Other suggestions.

A: Thanks for the kind reminder. We have modified the manuscript accordingly following the suggestions for better presentation.