A Large-scale Universal Evaluation Benchmark For Face Forgery Detection

Thanks for thoughtful and constructive feedback. We are glad to hear that you found the DeepFaceGen benchmark valuable. Below are responses to each of your comments. (The anonymous website referred to below is https://anonymous.4open.science/r/herE-CF34).

W1: a) Figures 1-3 lack clarity which fail to effectively highlight key findings; b) The use of small relative bar charts limits utility.

a) Thanks for valuable advice. Figures 1-3 present the overall performance on the benchmark. To better highlight findings, we will break these images into subfigures aligned with specific observations of each finding in the final version. For example, Finding 3, which examines detection difficulty based on the generation framework (autoregressive, diffusion, and GAN), will have separate performance plots for each category to better illustrate the underlying patterns.

b) Thank you for your valuable feedback. We will provide a table summarizing the detailed overall scores of the 20 detection models from Figures 1 and 2. Additionally, we have included tables showing the detailed performance of different forgery detection methods on various forgery types. Please refer to Section 1 of the anonymous website for the results.

W2: Findings lack rigorous support. Should be more cautious in their assertions and provide stronger empirical evidence.

Thanks for your insightful feedback. To validate the effectiveness of detail extraction modules, which are their commonalities across SOTA models performing well on both task-oriented and prompt-guided data, we conducted experiments involving multi-frequency feature analysis, multi-texture feature analysis, and multi-feature fusion.

Furthermore, we are actively working to provide more robust evidence for our experimental findings. Specifically, we have systematically reviewed all findings, revealing that the discoveries in DeepFaceGen stem from the inherent differences between task-oriented and prompt-guided data, as well as the unique challenges models face when detecting them. To explore this further, we conducted visual analyses across three dimensions: frequency domain, noise domain, and spatial domain. These analyses highlight distinct data characteristics and the varying effectiveness of detection strategies, establishing a clear connection between observations and conclusions. For detailed classifications, please refer to Section 2 of the anonymous website.

W3: a）The crucial issue of generalization capability is inadequately addressed. Figure 3's visualization of cross-generalization results is difficult to interpret, and the corresponding tables in Appendix F are poorly organized. b）Lack comparison with methods specifically designed for generalization.

a) Apologize for the difficulty in understanding. Figure 3 shows the results of cross-training and testing using the Xception model on image and video subsets, grouped by forgery method. The redder the block, the higher the metric; the bluer the block, the lower the metric. Appendix F validates the experiments by training 18 additional evaluation methods on the subset where Xception performs best and testing them on other subsets. To better highlight our findings, we will include a table in the final version summarizing generalization performance by forgery category, such as training on text-to-image data and evaluating on face-swapping data.

b) Thanks for valuable suggestions. Building on the generalization experiments within DeepFaceGen, we conducted cross-domain generalization tests.

Specifically, we explored the model's generalization when trained on DeepFaceGen and tested on non-face data (Genimage). We get the following findings: 1) Directly applying face AIGC to non-face AIGC presents substantial challenges. 2) Face detection models can adapt to identify non-face features when trained on similar generative frameworks. 3) Models focus primarily on facial forgery artifacts, which hinders their ability to generalize effectively. Detailed analysis can be found in Section 3 of anonymous website.

W4: Section 4.3 and Figure 4's feature visualization analysis is unclear, with the figure being difficult to distinguish.

Sincerely sorry for the confusion. Section 4.3 aims to visualize the feature distribution using T-SNE to examine whether the model can distinguish task-oriented and prompt-guided forgery features. To clearly show the feature distribution for each forgery type, Figure 4 presents the T-SNE results in the form of a "secondary directory" (refer to Table 2 in the appendix, lines 1189-1226). To better highlight Finding 10, we will include the distribution of task-oriented and prompt-guided types in the final version. For details, please refer to Section 4 on the anonymous website.

Thank you for your thoughtful feedback and for recognizing the strengths of our work. Below are our responses to W1-W4. (The anonymous website referred to below is https://anonymous.4open.science/r/3GRu-D654).

W1:Unclear and ambiguous definitions

Sorry for the confusion. Task-oriented-based face forgery summarizes the commonalities of traditional forgery methods, which are task-specific (e.g., face swapping, face alteration, and face reenactment). In contrast, prompt-guided-based face forgery focuses on the core technology of current AIGC generation techniques, which are driven by prompts (e.g., text2image, image2image, text2video). A detailed explanation and justification of this classification is provided below:

Due to the emergence of new AIGC forgery technologies, there is currently a lack of effective definitions distinguishing between "traditional" and "novel" methods. To clarify, we reviewed benchmarks that aim to differentiate between these two categories, and we found they typically rely on the following classification approaches:

• Model frameworks (e.g., GAN-based vs. Diffusion-based).
• Forgery regions (e.g., full-image generation, face reenactment, face swapping).
• Generation conditions (e.g., text2image, image2image, face swapping, face alteration).

For DeepFaceGen, which covers a comprehensive set of forgery methods, using the Model Frameworks and Forgery Regions approaches results in overlap between "traditional" and "novel" categories. Therefore, we categorize primarily by generation conditions. "Traditional" forgery methods focus on specific sub-tasks (e.g., face swapping, face alteration, and face reenactment), so we classify them as "Task-oriented." On the other hand, "novel AIGC" methods, despite differences in forgery region and model framework, are all prompt-driven (e.g., text2image, image2image, text2video), which is central to their free-form generation. Thus, we classify them as "Prompt-guided." The additional dataset you provided, Ref [1], supports this view and avoids overlap between "traditional" and "novel AIGC" data in this classification.

Thank you for your insightful question, which reflects your deep understanding of the field. Your feedback has prompted us to clarify the classification approach in the definition section and provide a detailed explanation in the appendix.

W2: a) Finding-1 and Finding-5 can be trivial; b) Previous works have already provided Finding-11. What is the significance of Deepfacegen；c) Finding-6: The same conclusion has been proposed by Ref[2].

a) Sorry for the confusion. Previous analyses focused on traditional forgery data. However, our findings in Finding-1 and Finding-5 also validate the effectiveness and patterns in novel forgery data, which helps advance the development of new forgery detection technologies in the field.

b) Sorry for the confusion. Finding 11 is derived from both traditional "Task-oriented" and newer "Prompt-guided" data, making it more universal and comprehensive. Previous methods provided only a rough analysis of task-oriented data in the frequency domain. In this work, we extend the analysis to prompt-guided data and uncover more detailed findings: mid-frequency features are more effective for detecting prompt-guided data, while high-frequency features perform better for task-oriented data.

c) Sorry for the misunderstanding. Ref[2] suggests that diffusion-based images outperform those generated by GANs and exhibit similar effects across various generation scenarios. Finding 6 highlights that current diffusion-based forged videos display similar forgery cues during detection.

W3: On the Incorrect Presentation of Finding 6

Thank you for pointing this out. We agree that detection performance doesn’t directly reflect the quality of forged data. Our intent was to show that, based on the performance of video forgery detection models on DeepFaceGen, existing diffusion-based techniques exhibit similar forgery artifacts, even as diffusion models evolve. We appreciate your careful review, and will clarify Finding 6 in the final version to avoid any misunderstandings.

W4:Reproduction Concerns

Thank you for your valuable suggestions. We have provided detailed experimental settings in the Evaluation Settings section and Appendix D.2. Additionally, we have maintained the open-source website to improve user access and usability of DeepFaceGen.

Ref[1]: Diffusion Facial Forgery Detection, ArXiv 2024.

Ref[2]: Diffusion Models Beat GANs on Image Synthesis, NeurIPS'21.

Thank you for your thoughtful feedback and for recognizing the strengths of our work. Below are our responses to Q1-Q5. (The anonymous website referred to below is https://anonymous.4open.science/r/3GRu-D654).

Q1: Meaning and rationale for forgery type definition.

Please refer to the answer of W1.

Q2: New findings?

Sorry for the confusion. Unlike previous studies based on traditional forged data, our findings are built on comprehensive evaluations covering both traditional and novel forgery methods. Based on this, we categorize the findings into two levels:

Generalization Validation on Novel Prompt-Guided Forgeries

• Finding 1 & 5: General findings derived from the commonalities of SOTA detection models.
• Finding 11, 12 & 13: Insights focused on forgery data detail extraction.
• Finding 2 & 7: Common forgery cues identified through the performance of detection models.
• Finding 8, 9 & 10: Generalization findings when models are tested across task-oriented and prompt-guided data.

Unique Findings on Prompt-Guided Data

• Finding 3 & 6: General findings on detection difficulty across different forgery frameworks and techniques.
• Finding 4: Insights into the impact of different input modalities on detection difficulty.
• Finding 14: Findings on how variations in facial attributes affect detection performance.

Detailed results can be found in Section 1 of the anonymous website. (We have also been conducting more in-depth experiments using DeepFaceGen. If interested, please refer to Sections 3-7 on the anonymous website.)

Q3: Recent datasets are missing from the discussion.

Thank you for pointing this out. We will include the dataset comparisons you provided in the RELATED WORKS section of the final version. Overall, the additional datasets you mentioned provide valuable contributions to the community, but they only cover a limited set of generation methods. There are significant limitations in data richness, modality diversity, and data fairness. For more details, please refer to Section 2 of the anonymous website.

Q4: Deep insights regarding the differences between image and video face forgery detectors?

Thanks for your valuable feedback. Image forgery detection focuses on spatial and frequency domain features:

• Spatial Domain: Analyzes distribution anomalies at the pixel or patch level.
• Frequency Domain: Identifies common forgery artifacts in forged versus real images.

Video forgery detection, with its richer temporal data, faces greater challenges in feature extraction. It emphasizes temporal correlations between frames to capture forgery anomalies. Despite their differences, image and video detection offer complementary insights, with many unexplored areas for cross-domain research. Here are our insights:

• Incorporating Temporal Information from Image Forgery Detection: In GAN and diffusion architectures, forgery detection based on reconstruction error has so far treated generative models merely as auxiliary tools. However, the generation process itself produces temporally varying image sequences. By delving into the principles of the forgery models, we can extract temporal cues during generation, potentially enhancing image-based detection.

• Prompt-guided Forgery Detection in Video: As highlighted in Appendix A.4, prompt-guided forgery data remains unexplored in video forgery detection. DeepFaceGen fills this gap by providing the necessary data support. Techniques from image forgery detection, such as noise reconstruction and contrastive learning between prompts and generated outputs, can offer valuable insights for detecting prompt-guided video forgeries.

• Introducing Graph Structures: Forgery detection in the field rarely explores graph structures. In other domains, such as spatiotemporal data, satellite navigation, and social network detection, graph structures have proven effective. Human face videos can essentially be viewed as dynamic graph structures, and constructing such structures is crucial for enhancing forgery detection.

Q5.Empirical observations. What are the underlying reasons behind observations?

Thanks for your valuable advice. We conducted detailed experimental analysis of the extraction module characteristics in SOTA models, resulting in Findings 11 and 12 (Appendix E). Furthermore, the findings in DeepFaceGen are derived from the differences between task-oriented and prompt-guided data, as well as the varying challenges models face when detecting them. To explore this, we conducted visual analyses from three perspectives: frequency domain, noise, and spatial domain. These analyses clearly reveal differences in the data itself and the effects of distinct detection approaches, establishing a strong connection between observations and conclusions. Detailed analysis is available in Section 8 of the anonymous website.

Thank you for your diligent review and comments. We are glad that you appreciated the comprehensiveness of our dataset and your recognition of our extensive experiments. (The anonymous website referred to below is https://anonymous.4open.science/r/UKvR-AE1E).

W1: The paper may not present a methodological contribution in itself.

We sincerely apologize for the confusion. With DeepFaceGen's comprehensive coverage of forgery data types (task-oriented and prompt-guided), we can expand or refine findings derived from earlier, more limited datasets. We have also conducted in-depth explorations using DeepFaceGen. Below are additional findings, which will be included in the final version.

• Capturing Temporal Forgery Artifacts in Diffusion-Generated Images: Introducing temporal information during the denoising process of diffusion improves forgery detection performance.

• Investigating the Impact of Identity Information on Face Forgery Detection Performance: Models that extract information directly from the data are affected by ID information, whereas reconstruction-based methods are not significantly influenced by ID information in their detection performance.

• Cross-domain Detection Performance from Face to Non-face Scenarios: (1) Directly applying face AIGC to non-face AIGC presents substantial challenges. (2) Face detection models can adapt to identify non-face features when trained on similar generative frameworks. (3) Models focus primarily on facial forgery artifacts, which hinders their ability to generalize effectively.

• Exploring Symmetry Differences Between Real and Prompt-Guided Data：Focusing on symmetry differences could help capture the physiological discrepancies between real and prompt-guided generated data, providing a new approach for image/video face forgery detection.

• The impact of various architectures on forgery detection：Through extensive comparisons of CNN-based and Transformer-based detection models， we found that in the video forgery detection domain, Transformer-based models generally outperform CNN-based models.

• Exploring data augmentation methods for improved detection performance: We explored the impact of pseudo-blending-fake data on detection performance using DeepFaceGen. Through extensive comparisons of model performance on both domain-specific and cross-dataset tasks with and without the use of pseudo-blending fake samples, we observed consistent improvements in detecting face-swapping forgeries.

Detailed analysis is available in Sections 1-6 of the anonymous website.

W2: a) The authors did not clarify the representativeness of 20 detection methods.b) the chosen methods seem to focus more on detection models that utilize frequency domain features.

a）Sorry for the confusion. In Appendix D (EVALUATION DETAILS), we provide an overview of the 20 evaluation methods, detailing their key characteristics, such as the technical approach and the types of forgery data they are suited for. We will provide more detailed descriptions in the main body of the final version. A summary table showcasing the representativeness of the 20 detection methods is provided in Section 7 of the anonymous website.

b) Thank you for your valuable advice. Using forgery detection models with different approaches indeed provides more comprehensive and reliable results. We have added F3Net , EfficientNet and RECCE. As a result, the detection methods in the "Details on Extraction Module" now cover naive detector, frequency domain, spatial domain, and pre-trained model-based feature extraction methods. The additional experiments show that using mid-frequency features significantly improves detection for prompt-guided types, while high-frequency features enhance detection for task-oriented types. Furthermore, texture features improve detection performance for both prompt-guided and task-oriented types, consistent with the validation experiments in the main text. More experimental results can be found in Section 8 of the anonymous website.

Thanks for your diligent review and comments. We appreciate your recognition of the value in addressing limitations of previous datasets. Below are responses to each of your comments. (The anonymous website referred to below is https://anonymous.4open.science/r/G6No-E3DF).

W1: a)13 findings are not impressive enough；b) Finding 1 simply reasserts well-known motivations from MAT. Finding 2&7 are common sense; c) Recommend consolidating a Findings list in the conclusion.

a) Sorry for the confusion. DeepFaceGen fills the gap in the field by incorporating both novel and traditional forgery methods. This allowed us to uncover novel findings not possible with previous benchmarks. We have focused on using DeepFaceGen for further exploration in deepfake detection. Below are the additional experiments which will be included in the final version:

• Capturing Temporal Forgery Artifacts in Diffusion-Generated Images: Introducing temporal information during the denoising process of diffusion improves forgery detection performance.
• Investigating the Impact of Identity Information on Face Forgery Detection Performance: Models that extract information directly from the data are affected by ID information, whereas reconstruction-based methods are not significantly influenced by ID information in their detection performance.
• Cross-domain Detection Performance from Face to Non-face Scenarios: (1) Directly applying face AIGC to non-face AIGC presents substantial challenges. (2) Face detection models can adapt to identify non-face features when trained on similar generative frameworks. (3) Models focus primarily on facial forgery artifacts, which hinders their ability to generalize effectively.
• Exploring Symmetry Differences Between Real and Prompt-Guided Data：Focusing on symmetry differences could help capture the physiological discrepancies between real and prompt-guided generated data, providing a new approach for image/video face forgery detection.

Detailed results are in Sections 1-4 of the anonymous website.

b) Sorry for the confusion. MAT focuses on texture feature analysis for task-oriented data (Appendix E, lines 1382-1384). In contrast, we expand the analysis to include the novel prompt-guided data, revealing that mid-frequency features are more effective in detecting such data. Additionally, our approach incorporates Multi-frequency Feature Analysis and Multi-feature Fusion, going beyond just texture features.

To the best of our knowledge, there has been no prior study systematically comparing the quality of task-oriented and prompt-guided data before DeepFaceGen. Building on the 20 evaluation models on DeepFaceGen, we further tested data from different generation methods and frameworks, leading to Findings 2 & 7.

c) Thanks for your valuable advice. We will include a Findings list in the conclusion section of the final version to make it easier for readers to access our experimental results.

Q1: Various architectures (e.g., CNN vs. Transformers) or modalities (e.g., frequency vs. noise) have on deepfakes

We are impressed by the depth of your insightful comments.The following is our experimental exploration of the proposed aspects.

Various architectures: The use of different model architectures is undoubtedly valuable for the diverse development of deepfake detection. In fact, we are currently exploring the performance differences of various model architectures. Through extensive comparisons of CNN-based and Transformer-based detection models, we found that in the video forgery detection domain, Transformer-based models generally outperform CNN-based models.

Various modalities：Through detailed experiments on frequency-based and noise-based methods, we observed the following: 1) Frequency-based methods struggle to capture the forgery fingerprints of prompt-guided data. 2) Both noise-based and frequency-based methods primarily detect fingerprints specific to particular forgery frameworks, but these fingerprints lack generalizability across different frameworks.

More experimental results are in Sections 5-6 of the anonymous website.

Q2: Beneficial data augmentations or training strategies

We conducted detailed experimental analyses from the perspectives of Geometric Transformations, Color and Intensity Adjustments, and Image Quality Transformations, including methods such as Rotation, Isotropic Resize, Random Brightness and Contrast, FancyPCA, Hue Saturation Value (HSV) Adjustment, Image Compression, and Centrifugal, Additive, and Affine Transformations. The experimental results demonstrate that the following two methods significantly improve detection model performance: (1)Training the networks on local patches,cropped from the image with no resizing.(2)Centrifugal, additive, and affine transformations as data augmentation.

More experimental results can be found in Section 7 of the anonymous website.