DiffusionFake: Enhancing Generalization in Deepfake Detection via Guided Stable Diffusion

We sincerely appreciate your positive acknowledgment of our method's rationality, as well as the meaningful questions you've raised. Below are our responses to your specific inquiries:

Q1: Concern about weight module.

Thanks for your question. We agree that Stable Diffusion's latent representation is non-linear. However, using cosine similarity of non-linear features to measure data relationships is common, especially in face recognition where non-linear model features are often used for similarity comparisons. Moreover, Stable Diffusion's paper[1] notes that the compression latent space preserves input details and allows good reconstruction. Thus, we argue that similarities in this compressed space can represent original image similarities, serving as a supervision signal for our weight module.

To support this assertion, we direct your attention to Figure 7 in our main paper. This figure demonstrates that the trained weight module derived from our method largely aligns with intuitive expectations of image similarity. This alignment suggests that, despite the non-linear transformation, our approach captures meaningful relationships between images.

We acknowledge that there is room for more rigorous assumptions in future work. Potential improvements could include using non-linear space metric functions or simpler, direct image similarity metrics such as SSIM or PSNR.

We appreciate your insightful comment, which has highlighted an important area for future refinement in our approach.

[1] High-Resolution Image Synthesis with Latent Diffusion Models.

Q2: Influence of the loss weight

We appreciate your observation regarding the loss weights in Equation 13. To address this, we conducted comprehensive ablation studies to determine the optimal values for $\lambda_s$ and $\lambda_t$.

Following the ControlNet setup and considering that target reconstruction is relatively stable, we initially fixed $\lambda_t$ at 1 and varied $\lambda_s$ through values of 0.1, 0.3, 0.5, 0.7, and 1.0. Our experiments showed that $\lambda_s$ = 0.7 yielded the best average performance across five test datasets. The quantitative results are shown in the table below:

We then fixed $\lambda_s$ at 0.7 and conducted ablation studies on $\lambda_t$, finding the peak performance at $\lambda_t$ = 1. The experimental results are summarized in the following tables:

These results align with our intuition. The source image often differs significantly from the fake image, so a slightly smaller loss weight for the source $\lambda_s$ helps maintain training stability. We observed that if $\lambda_s$ is too large, the loss becomes difficult to minimize.

We will include this detailed analysis in our revised manuscript to provide a clearer understanding of our model's behavior and optimization.

Q3: Concern about cross-model evaluation.

We appreciate your question. As mentioned in lines 246-250 of our paper, our method primarily aims to enhance model generalization. Consequently, the majority of our experiments are cross-dataset evaluations.

In Table 1, following the protocols established in papers such as DCL[2] and DeepfakeBenchmark[3], we train our model on the FFpp dataset and evaluate it on other test sets. This approach ensures that the model is tested on samples generated by methods not seen during training.

Furthermore, we have included additional cross-manipulation evaluation experiments in the Appendix (kindly refer to Table 4 and Table 5), which test attack types different from those used during training.

We acknowledge that we could have made this experimental setup clearer in the main text. In our revised manuscript, we will provide a more explicit explanation of our cross-dataset and cross-model evaluation strategy to avoid any confusion.

[2] Dual contrastive learning for general face forgery detection.

[3] DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection.

Q4: Concern about high resolution image.

Our experimental setup follows the protocols established by standard setting used in many previous works, where images are uniformly resized to 224x224 for both training and testing. Following your suggestion, we conducted additional experiments with a higher resolution of 384x384 and 512x512. The results, as shown in the table below, demonstrate that our method still achieves improvements at this higher resolution:

These findings indicate that our approach remains effective for larger image resolution.

Furthermore, we have tested our method on high-quality images using the DiffSwap dataset, which consists of high-resolution images (1024x1024) processed through diffusion-based face swapping. Our method showed a relative improvement of approximately 6% on this dataset, further demonstrating its generalization capability under high-quality, high-resolution conditions.

In our revised manuscript, we will include these additional experiments to provide a more comprehensive evaluation of our method's performance across various image resolutions and quality levels. Thank you for this valuable suggestion, which has helped us to more thoroughly validate our approach.

Dear Reviewer NDSD,

Thank you for your invaluable efforts and constructive feedback on our manuscript. We greatly appreciate your positive evaluation of our paper's rationale. We have endeavored to provide comprehensive responses to the concerns you raised in your review.

As the discussion period draws to a close, we eagerly await your thoughts on our response. We sincerely hope that our revisions align with your expectations. If there are any remaining concerns or aspects that require clarification, we are ready to address them as soon as possible.

Best regards,

The Authors

We sincerely appreciate your positive evaluation of our paper, particularly your comments that our work is "logically sound," "novel," and "easy-to-follow." We are committed to further improving our manuscript based on your valuable suggestions. Below, we address each of your questions in detail:

Q1: Concern about the same ground truth.

Thank you for this excellent question. Indeed, as the FFpp dataset contains four different attack methods, there are instances where different samples have the same reconstruction ground truth. However, the reconstruction weights differ significantly in these cases, which is one of the reasons we designed the weight module.

Figure 7 in our paper demonstrates the varying weights corresponding to different forgery types. As shown, even when the ground truths are identical, the dependence on source and target conditions differs. This variation leads to diversity during training.

We appreciate you bringing this to our attention. We will include this discussion in the main text of our revised paper to provide a clearer explanation of how our method handles this scenario.

Q2: Concern about quality of reconstruction.

We appreciate your perceptive question regarding the quality of reconstruction. Our method takes on the unique challenge of reconstructing source and target images from fake images, which inherently involves significant information loss. This loss can lead to inaccuracies in expression or blurring, as you noted in Figure 3A of our main paper.

It's important to emphasize that the primary aim of our DiffusionFake framework is not perfect reconstruction, but rather to compel the detection model to extract source-related and target-related features. This extraction process enhances the model's generalization capabilities, which is our ultimate goal. The quality of reconstruction serves as a means to this end, rather than being the end itself.

Interestingly, we've discovered that the input noise significantly influences the fine-grained expression control in reconstructed images. To illustrate this, we've included Figure 2 in our rebuttal PDF. This figure shows target images reconstructed using five different noise patterns, along with their corresponding PSNR and SSIM scores relative to the target ground truth. The final noise pattern, in particular, yields images that closely match the target ground truth in both expression and detail.

Q3: Concern about ablation studies.

Thank you for your question. Following your suggestion, we conducted ablation studies on our feature-transform module. Specifically, we ablated three components: space attention, channel attention, and the self-attention module used for fusion. The results are shown in the table below:

As we can observe from the table, both space attention and channel attention show improvements compared to not using them. This indicates that feature selection in both dimensions is crucial for the results. Additionally, we found that using self-attention for fusion performs better than direct fusion. This is because the cross-attention operation allows for more comprehensive feature integration. Finally, we discovered that the combination of all three components achieved the best performance.

Q4: Question about unfrozen Stable Diffusion.

Thank you for your question. Following your suggestion, we attempted to fine-tune the decoder part of the Stable Diffusion model during training. The results are shown in the table below:

We observed that while training time and GPU memory usage increased, the performance improvement was not significant. This may be because the pre-trained knowledge in the Stable Diffusion model is vast, and the dataset for this task is insufficient to adjust the entire Stable Diffusion model effectively. Moreover, our primary goal is to force the encoder to extract source-related and target-related features to improve generalization through reconstruction. Therefore, fine-tuning the Stable Diffusion part doesn't provide substantial benefits.

We appreciate your suggestion as it has led to these valuable insights. This experiment has helped us better understand the role and limitations of fine-tuning in our approach.

Q5: Concern about impact of Stable Diffusion.

Thank you for your insightful question. Due to time constraints, we experimented with two pre-trained SD models. First, we used a randomly initialized diffusion model without SD pre-training as a baseline. Second, we employed the SD 2.1 model for training. The results are shown in the table below:

As we can observe, without using the pre-trained SD model, information loss cannot be adequately compensated, leading to unsuccessful reconstruction. This results in unstable training and minimal performance gains.

On the other hand, using the latest SD 2.1 model slightly outperforms SD 1.5. This improvement may be attributed to SD 2.1's enhanced capabilities, which provide the DiffusionFake framework with stronger information recovery abilities. Consequently, this allows the encoder to extract more generalizable features.

Dear Reviewer b2gp,

We greatly appreciate your valuable feedback and the time you've taken to review our manuscript. We are sincerely grateful for your positive evaluation of our paper. Additionally, we would like to express our appreciation for the constructive experimental suggestions you've provided, which will undoubtedly make our paper more comprehensive.

As we approach the end of the discussion period, we eagerly await your thoughts on our response. We sincerely hope that our revisions align with your expectations. If there are any remaining concerns or aspects that require clarification, we are ready to address them as soon as possible.

Best regards,

The Authors

We sincerely appreciate your recognition of our work and your valuable suggestions. Your feedback is precious to us, and we will carefully revise our paper based on your comments, especially addressing the references you mentioned and correcting any capitalization issues. Regarding your specific question, we offer the following response:

Q1: Insufficient comparative discussion:

Thank you for raising this point. We agree that the papers you mentioned are indeed significant contributions to this field. In our revised paper, we will thoroughly compare and discuss these articles, clearly articulating the differences and connections between our method and the approaches presented in these works. This will provide a more comprehensive context for our research within the current state of the field.

Q2: Concern about inconsistent results.

We appreciate you highlighting this important point. The discrepancies in experimental results arise from varying implementation details across studies. These differences include face extraction methods, cropping ratios, frame sampling, augmentation techniques, and testing protocols. Additionally, many papers lack results for certain test sets like DiffSwap and Wild Deepfake.

To ensure a fair comparison, we retrained all models with publicly available code under identical conditions. We strictly adhered to the original hyperparameters and settings specified in the open-source code for each reimplemented algorithm, ensuring a faithful performance comparison.

In our revised paper, we will provide a detailed explanation of our experimental setup and clearly indicate that the reported results are from our reimplementation under consistent conditions. We will also make the training configurations for other methods publicly available in our open-source code.

Q3: Visualization about misprediction results.

We appreciate your suggestion for a deeper analysis of our method's performance. Upon examining our misprediction results, we identified two main categories of errors, as illustrated in Fig. 4 of rebuttal PDF:

1). Profile view images: These images present a challenge during training, as they are difficult to reconstruct into source and target images due to significant information loss. This results in misclassification during inference. 2). Low-quality images: Our method encourages the detector to decouple source-related and target-related features to improve generalization. However, low-quality, blurry images hinder the network's ability to extract these features effectively, leading to misclassification.

In future work, we will focus on optimizing these two types of images, such as increasing the weight of low-quality data reconstruction during training, and using data augmentation to supplement the side faces in the training data.

We will include this analysis and the corresponding visualizations in our revised paper to provide a more comprehensive understanding of our method's strengths and limitations

Dear Reviewer VbH6,

We are deeply grateful for your thorough review and insightful comments on our paper. We greatly appreciate your evaluation of our work and assure you that we will comprehensively incorporate your feedback into our revised paper, enhancing its content and quality.

As the discussion period draws to a close, we eagerly await your thoughts on our response. We sincerely hope that our revisions align with your expectations. If there are any remaining concerns or aspects that require clarification, we are ready to address them promptly.

Best regards,

The Authors

We sincerely appreciate the reviewer's recognition of our work. Your positive feedback is greatly encouraging. We are grateful for your careful consideration of our rebuttal and the time you've invested in evaluating our research.

We sincerely appreciate your positive feedback on our paper, particularly your comments that our work is "well-motivated" and "simple and intuitive". We are committed to further improving our manuscript based on your constructive suggestions. Below, we address each of your questions in detail:

Q1: Question about DFD dataset performance:

As you noted, the DFD dataset indeed shows higher performance compared to other datasets. Our analysis reveals that DFD images exhibit noticeable inconsistencies, as shown in Fig. 1 of the rebuttal PDF, which closely resemble the inconsistency patterns in the FFpp training set. This similarity stems from using the same forgery method and post-processing.

Furthermore, UIA-VIT performs exceptionally well on the DFD dataset (94.68%). This is because UIA-VIT leverages an Unsupervised Patch Consistency Learning module, which effectively models these inconsistencies by learning forgery location maps between real and fake image pairs. This approach is particularly effective for detecting inconsistencies similar to those in the training set.

Moreover, our detector uses the simplest EN-B4 and VIT-B models without any special design. Even in this scenario, it shows significant relative improvements without increasing parameters or inference time. Compared to baselines, we achieved a 4.14% improvement over En-b4 and a 5.98% improvement with VIT-B backbone on the DFD dataset. Moreover, on other higher-quality datasets, our method outperforms UIA-VIT by about 4% without increasing parameters.

Q2: Concern about expression reconstruction:

Thanks for your insightful question. Our approach uniquely reconstructs source and target images from fake ones, facing challenges due to information loss. This can lead to expression inaccuracies or blur, as seen in Figure 3A of the main paper. However, DiffusionFake's primary goal is to compel the detection model to extract source-related and target-related features, enhancing generalization. Reconstruction quality serves as a means to this end, not the ultimate objective.

Moreover, we've observed that fine-grained expression control in reconstructed images is closely related to the input noise. With suitable input noise, we can achieve better reconstruction results. In Fig 2 of the rebuttal PDF, we visualize target images reconstructed under five different noise patterns, along with their PSNR and SSIM scores compared to the target GT. Notably, the last noise pattern produces images with expressions and details closely matching the target GT. This finding provides valuable insights into our method's capabilities and potential for improvement. We will add the results in the revised version.

Q3: Question about reconstruction metrics and performance:

We appreciate your suggestion on reconstruction quality metrics. Following your recommendation, we calculated PSNR and SSIM for each model in the ablation study from Table 2. For each model, we used 10 random noise sets and their corresponding target GT, then averaged the values. The results are shown below:

Our analysis reveals a positive correlation between reconstruction quality and detection performance. Models with better reconstruction quality generally demonstrated higher detection accuracy. Notably, when the SD pre-trained model is not used, the generation quality is very poor, corresponding to significantly worse results. This finding supports the intuition that better reconstruction ability contributes to more effective feature extraction, which in turn leads to improved detection performance. We will add this results in the revised paper.

Q4: Comparison with generative methods:

Our work is not the first to use generative models for face forgery detection. Previous methods, such as RECCE, have used Auto-Encoder approaches to improve detector generalization by reconstructing real samples.

Our approach differs in motivation and reconstruction targets. Previous methods typically reconstruct samples directly to learn data distribution without information loss. In contrast, our motivation is to recover source and target images from fake images, enhancing the detection model's ability to capture decoupled features. This task is challenging due to the significant disparity between fake and original counterparts, often resulting in substantial information loss. We address this by incorporating a pre-trained Stable Diffusion model to compensate for lost information.

However, traditional generative methods like RECCE, using simple Auto-Encoder architectures, cannot handle such complex reconstruction tasks. Fig. 3 in our rebuttal PDF compares RECCE's and our method's reconstruction of source and target images. We can observe that RECCE fails to reconstruct due to information loss, while our method, leveraging the SD model, achieves better restoration. This demonstrates the key difference between our approach and other reconstruction-based methods.

We'll include these results and visualizations in our revised paper.

Q5: Concern about "Deepfake" Term.

We appreciate your insight regarding the term "Deepfake." We agree that the scope of Deepfake techniques has expanded beyond simple face swapping. To avoid confusion, we will clearly specify the applicable range of our method in the main text of our paper.

Dear Reviewer tkET,

We are deeply grateful for your thorough review and insightful comments on our paper. Your positive evaluation has been highly encouraging. We also appreciate your constructive suggestions, which have significantly improved our paper's depth and completeness.

As the discussion period draws to a close, we eagerly await your thoughts on our response. We sincerely hope that our revisions align with your expectations. If there are any remaining concerns or aspects that require clarification, we are ready to address them promptly.

Best regards,

The Authors