Orthogonal Subspace Decomposition for Generalizable AI-Generated Image Detection

We sincerely thank Reviewer SVkD for the constructive comments, insightful questions, and useful suggestions. We address the reviewer's concerns below.

Q1. The authors should provide additional visualizations for different testing datasets. For instance, face deepfakes might be less diverse than general AI-generated images, leading to inconsistent results.

R1. Thanks for your valuable suggestion.

• Indeed, the number of principal components (effective rank) varies across datasets because our method measures the "effective dimension" of the feature space, which depends on the input data distribution.
• To validate this, we analyzed the effective rank on multiple test datasets, including CDF-v2 (face-focused) and UniversalFakeDetect (general-content). Key observations from Table 5 are:
  • General datasets exhibit a higher effective rank than face-specific datasets. This is because general datasets contain diverse objects, leading to a more complex feature space.
  • Our SVD-based method outperforms LoRA and full fine-tuning (FFT) in capturing a higher-rank feature space, preserving more discriminative information.

Table 5: Evaluation results on different testing datasets.

---

Q2. The authors have not provided a clear explanation for why n-r=1 achieves the best results.

R2. Thanks for the interesting questions. Our choice of using a lower rank (specifically, n-r=1) for fine-tuning DFD is primarily motivated by two critical factors:

• First, the nature of the real-fake classification task itself makes it relatively straightforward. Specifically, fake samples in existing training sets tend to exhibit a limited number of distinctive forgery patterns (FF++ contains only four forgery types), each with relatively simple and consistent characteristics.
  • Due to this simplicity and limited diversity, a low-rank adaptation with a small rank (e.g., "n-r"=1, 4, or 16) is sufficient for the model to effectively learn these forgery patterns. As demonstrated by Table 5 in our paper, choosing ranks of 1, 4, or 16 yields very similar performance results. Given this observation, we prioritize efficiency and parameter economy, making rank 1 the optimal choice.
• Second, the inherent characteristics of binary classification further justify selecting a smaller rank. Binary classification tasks typically do not require the model to learn extensive and nuanced patterns, but rather to identify just enough distinctive features to separate the two classes, making the learned feature space inherently constrained.
  • To illustrate, when distinguishing between cats and dogs, the classifier might achieve good accuracy simply by examining the tails, without needing detailed knowledge about specific breeds such as Corgis or Shibas. Thus, binary classification inherently simplifies the complexity of the learning problem, meaning that employing a higher rank would not provide significant additional benefit.
  • In contrast, more complex tasks like multi-class classification (as discussed in our extended framework proposed in our response R1 of the Reviewer cn1X) necessitate a higher rank (the value of "n-r"). Indeed, our experiments indicate that increasing the value of "n-r" in multi-class scenarios leads to improved performance, highlighting the relationship between task complexity and optimal rank selection.

Table 6: Optimal rank selection based on the task complexity. We evaluate all the models on the Chameleon dataset.

---

Q3. Many details in the analysis are missing, such as Figure 5 and Figure 6.

R3. Thanks for the kind mention. All analysis figures, including Figure 5 and Figure 6, are tested on FF_c23 (test), which is commonly used as the within-domain testing dataset. We will clarify it in the revision.

---

Q4. The implementation and experimental details of GenImage are missing.

R4. Thanks for your kind mention. We will add the implementation details and clarify them clearly in the revision.

We sincerely thank Reviewer j2Rb for the constructive comments, insightful questions, and useful suggestions. We greatly appreciate and are encouraged by the reviewer's recognition of our insightful and in-depth analysis, methodological novelty, and comprehensive experiments with high generalization performance. Additionally, the reviewer raised several important concerns and questions, which we address in detail below.

Q1. The paper does not provide an intuitive visualization of residual and principal components using PCA. It would provide some new insights and findings.

R1. We genuinely appreciate the suggestion. We perform the visualization of the attention maps before and after our proposed orthogonal training on the UniversalFakeDetect dataset. Specifically, for each block of the vision transformer, we visualize the attention map, which represents the attention coefficient matrix calculated between the cls token and the patch tokens.

• Specifically, the attention map is computed across the multiple heads on average and is presented alongside the principal weights, residual weights, and total weights, respectively.
• We have three discoveries in general:
  • 1. The attention map of principal weights is almost identical to the attention map of the total weights for each block;
  • 2. Before and after training, the attention maps of the residual weights in the earlier blocks do not respond (e.g., for ViT-L of CLIP model, the first 22 blocks do not respond);
  • 3. Only the attention maps in the last blocks of the residual weights respond, which means they contain the real/fake discriminating information (e.g., for ViT-L of CLIP model, the last 2 blocks respond).

Following the reviewer's suggestion, we will add these visualization results and the corresponding analysis into our revision.

---

Q2. Some important implementation details of fine-tuning and evaluations are missing in the paper. For instance, how many layers are fine-tuned? What kinds of data augmentations are used in this paper?

R2. Thanks for the suggestion. We list the implementation details below.

• Fine-tuned layers: By default, we fine-tune all query (Q), key (K), value (V), and output (Out) components across every layer of the given architecture, such as CLIP.
• Data augmentation strategy: We adopt standard data augmentation techniques commonly used in each specific benchmark. For instance, we follow [1, 2] for deepfake detection, [3] for the UniversalFakeDetect benchmark, and [4] for the GenImage benchmark.

Following the reviewer's suggestion, we will present a detailed description in the revision.

[1] LSDA. CVPR 2024. [2] DeepfakeBench. NeurIPS 2023. [3] UnivFD. CVPR 2023. [4] NPR. CVPR 2024.

---

Q3. The observation in Table 6 that lower-rank SVD achieves better generalization results is intriguing. The authors should provide a detailed explanation for this.

R3. We genuinely appreciate the suggestion. Our choice of using a lower rank (specifically, "n-r"=1) for fine-tuning DFD is primarily motivated by two critical factors:

• First, the nature of the real-fake classification task itself makes it relatively straightforward. Specifically, fake samples in existing training sets tend to exhibit a limited number of distinctive forgery patterns (FF++ contains only four forgery types), each with relatively simple and consistent characteristics.
  • Due to this simplicity and limited diversity, a low-rank adaptation with a small rank (e.g., "n-r"=1, 4, or 16) is sufficient for the model to effectively learn these forgery patterns. As demonstrated by Table 5 in our paper, choosing ranks of 1, 4, or 16 yields very similar performance results. Given this observation, we prioritize efficiency and parameter economy, making rank 1 the optimal choice.
• Second, the inherent characteristics of binary classification further justify selecting a smaller rank. Binary classification tasks typically do not require the model to learn extensive and nuanced patterns, but rather to identify just enough distinctive features to separate the two classes, making the learned feature space inherently constrained. Thus, binary classification inherently simplifies the complexity of the learning problem, meaning that employing a higher rank would not provide significant additional benefit.
  • In contrast, more complex tasks like multi-class classification (as discussed in our response R1 of the Reviewer cn1X) necessitate a higher rank (value of "n-r"). Indeed, our experimental results indicate that increasing the value of "n-r" in multi-class scenarios leads to improved performance, highlighting the relationship between task complexity and optimal rank selection.

---

Q4. The improvement of the two proposed constrained losses shown in the ablation study is limited.

R4. Thanks for your comment. Please refer to our response of R4 of the reviewer 4w2k.

We sincerely thank Reviewer 4w2k for the constructive comments, insightful questions, and useful suggestions. We greatly appreciate and are encouraged by the reviewer's recognition of our motivation with sufficient and reasonable evidence, methodological novelty, and interesting analysis method. Additionally, the reviewer raised several important concerns and questions, which we address in detail below.

Q1. What about the performance of ViT models trained on ImageNet-1K and other ViT-based VFMs?

R1. Thank you for raising this. Following the reviewer's suggestion, we conduct additional experiments comparing ViT models pre-trained on ImageNet-1K and SigLIP using different adaptation methods. Results are summarized clearly in Table 2 below:

Table 2: Results of other ViT-based baseline models.

From these results, we observe that our SVD-based adaptation method consistently achieves the best performance over the LoRA-based adaptation and full fine-tuning (FFT) across different pre-training models.

---

Q2. How can the authors implement their method in GenImage benchmark?

R2. We sincerely appreciate the kind comment. Our experiments primarily follow the implementation details and experimental settings described in recent SOTAs [1, 2]. This alignment ensures consistency and transparency, making our results directly comparable with existing studies.

[1] NPR. CVPR 2024. [2] FatFormer. CVPR 2024

---

Q3. Also, can the detection method be applied to detect more complex and advanced fakes, such as the talking-head generation contents?

R3. Thank you for your question. Following your suggestion, we have extended our evaluation to include highly realistic talking-head deepfake content, i.e., HeyGen, selected from the DF40 dataset. To ensure a fair comparison, we add four SOTA detection methods under identical experimental conditions for a fair comparison.

The results, summarized clearly in Table x below, demonstrate that our method achieves superior performance compared to the latest SOTA detectors, even on advanced commercial talking-head manipulations.

Table 3: Evaluation results on the advanced talking-head generation fake contents.

From the results above, we find that some detectors trained on the face-swapping data fail to generalize the talking head contents. However, most previous works are conducted using only the face-swapping deepfake data. Inspired by the reviewer's comment, we plan to add this additional evaluation to the revision and further enlarge our evaluation in the future.

---

Q4. The authors did not conduct an ablation study on the weights of the orthogonality loss and singular value loss. How are these weights allocated in the loss function?

R4. Thanks for the comment. In our experiments, we set the weights for both the orthogonality loss and singular value loss terms to 1.0 by default. To further explore the influence of these loss terms, we adjust each hyper-parameter across a wide range.

As shown in Table 4 below, varying these parameters results in only slight fluctuations in performance, indicating the stability of our method. Importantly, the inclusion of these loss terms consistently improves performance compared to models without them.

Table 4: Ablation studies regarding different weights of the loss terms.

---

Q5. I hope the authors can cite references [1] and [2] and include a brief discussion on their relevance.

R5. Thanks for the valuable suggestion. Both [1] and [2] introduce the concept of effective rank to quantitatively measure the dimensionality of the feature space, which is similar to our case. We will provide a detailed discussion in our revision. Thank you again.

We sincerely thank Reviewer cn1X for the constructive comments, insightful questions, and useful suggestions. We greatly appreciate and are encouraged by the reviewer's recognition of our motivation, thorough experimental and theoretical analysis, methodological novelty, and superior experimental performance. Additionally, the reviewer raised several important concerns and questions, which we address in detail below.

Q1. The paper treats all forgery methods as a single category during training. Has the consideration of the specificity and generalization of different forgery methods been explored?

R1. Thanks for your very insightful question. As highlighted by the reviewer, treating all forgery methods as a single category in binary classification may potentially risk losing specificity and generalization—a concern we have not yet verified.

However, please note that this limitation is inherent to binary classification tasks in general, rather than specific to our proposed SVD-method in this work (our SVD-based approach is designed specifically for adapting to new forgery types while preserving pre-trained knowledge).

Following the reviewer's suggestion, we plan to propose an extended version (multi-task) based on our current binary framework, aiming explicitly at enhancing the balance between specificity (handling known forgeries effectively) and generalization (detecting unknown or unseen forgeries).

The preliminary idea of our future multi-task learning framework features:

• (1) Dual-head structure: Two separate heads are employed—one dedicated to multi-class classification (specific head) and another for binary classification (general head).
  • Specific Head (multi-class): Learns fine-grained differences among known forgery types, focusing on fitting to the training distribution (specificity, IID).
  • General Head (binary): Captures shared features across forgery types, enabling the detection of unseen or novel manipulations (generality, OOD).
• (2) Adaptive Dynamic Inference: Combining predictions from both heads based on confidence, balancing specificity and generality dynamically during inference.

Specifically, our dynamic inference strategy is as follows:

• Compute the prediction probability (i.e., confidence) according to the specific head's logits.
• If the maximal prediction confidence across classes is below a threshold (indicating uncertainty or a "flat" distribution), then making decision based on the binary general head, otherwise based on the specific multi-class head.

This adaptive strategy effectively maintains specificity for known forgery methods while providing robust generalization against unseen or evolving forgeries.

Due to the limited content in the rebuttal, we cannot provide exhaustive experimental details at this stage. But following the reviewer's suggestion, we evaluate our binary model and multi-task model on the $\text{Chameleon}$ dataset (see R2 below for details).

---

Q2. Chameleon is a high-quality dataset. Testing on this dataset would further validate the generalization ability.

R2. Thanks for introducing and highlighting the high-quality and challenging dataset, $\text{Chameleon}$. Following the suggestion, we have conducted additional evaluations using this dataset. We follow the proposed setting in the original paper for experiments, i.e., training on GenImage (whole) and testing on $\text{Chameleon}$.

As shown in Table 1 below, our method achieves superior generalization performance compared to baseline methods and other SOTA detectors. Also, the proposed extension version, i.e., multi-task framework, further boosts and refines the results, alleviating the potential loss of specificity and generality problem raised by the reviewer.

Table 1: Evaluation results on Chameleon. All detectors are trained on the GenImage dataset.

Furthermore, all evaluated methods experience a notable performance decline on $\text{Chameleon}$, highlighting the dataset's significant difficulty. Inspired, we plan to conduct an in-depth analysis of $\text{Chameleon}$ in future research. Again, we greatly appreciate your suggestion.

---

Q3. About the robustness evaluation.

R3. Thank you for your question. We have already acknowledged this concern and have performed a robustness evaluation on the deepfake detection benchmark to assess our model's robustness, as presented in Figure 7 of our appendix. Following [1,2], we evaluate three types of image degradation: block-wise distortion, contrast changes, and JPEG compression. This experiment confirms the model's robustness against various perturbations.

[1] LSDA, CVPR 24.

[2] LipForensics, CVPR 21.