VLForgery Face Triad: Detection, Localization and Attribution via Multimodal Large Language Models

Response to Weakness 1: Differences Between VLF and Prior Manipulation Detection Datasets

Thank you for this important comment. At the early stage of our work, we indeed conducted a careful survey of existing datasets such as [1] and others. However, we found that they could not fully meet the needs of our task. Below, we outline key differences and contributions of the proposed VLF benchmark:

1. Prompt Design for Face Generation:
   Existing AIGC datasets (e.g., [1], [2]) often use short and simplistic prompts, which may not produce low-quality images, but result in less detailed and diverse facial features.
   In contrast, we design long-format prompts that are*specifically tailored for face generation, inspired by dedicated YouTube channels studying facial synthesis (details in Section B.3). Our prompts include semantically rich tokens such as [framing], [camera_angle], [lighting], etc., which significantly enhance facial realism and diversity.

2. Generator Quality and Coverage:
   [1] often rely on privately trained or lower-quality open-source models, which may not match the realism of widely-used diffusion models like Flux, Kandinsky, etc.

3. Detailed Annotations for Partial Synthesis:
   Our task focuses on fine-grained forgery localization and attribution, which requires precise metadata. Existing datasets rarely provide such details.
   In VLF, we include annotations on:

   • Whether edits involve single-region or multi-region manipulation;
   • The exact name and location of modified regions;
   • Whether the image was synthesized from scratch or by editing real images.

• [1] Diffusion Facial Forgery Detection
• [2] GenImage: A Million-Scale Benchmark for Detecting AI-Generated Image

Response to weakness 2: Cross-Dataset Evaluation on Challenging Benchmarks (e.g., DFDC, FF++)

We appreciate the reviewer’s suggestion regarding broader cross-dataset evaluation. We agree that the current version of our framework has not been extensively evaluated on traditional Deepfake datasets, which could limit the generalizability of our conclusions.

Due to time and computational resource constraints, we were unable to build and run a dedicated training/testing pipeline on large-scale Deepfake datasets like DFDC. However, as an initial step, we conducted evaluation on the face-swapping subset of FaceForensics++.

Table: Evaluation on Traditional Face-swapping Deepfakes type. Results on different training and testing subsets using VLForgery. Accuracy is used for evaluation.

We will include these results in the revised version of the paper and plan to expand to additional benchmarks in future work.

Response to Weakness 3: Limited Baselines – Consideration of Recent Deepfake Detection Methods [4, 5, 6]

Thank you for this valuable comment. We understand your concern regarding the comparison with only a few baseline methods. Below, we clarify our rationale:

Although both traditional Deepfake datasets (e.g., [FF++]) and AIGC-based datasets (e.g., [1]) focus on face manipulation, the manipulation mechanisms differ fundamentally. Traditional Deepfakes are typically generated via face-swapping or reenactment techniques such as StarGAN, FaceAPP, or FaceSwap, whereas our dataset, VLForgery, is based on diffusion models and prompt-driven generation.

Methods like [4, 5, 6] enhance forgery detection by exploiting artifacts specific to face-swapping (e.g., warping, boundary inconsistencies). While these are effective for traditional Deepfakes, such artifacts do not exist in AIGC-based forgeries. Instead, recent works for AIGC forensics (e.g., DRCT using self-reconstruction, SAFE and CNNspot using traditional image augmentation, and SRM, SPSL, NPR employing frequency-based features) are more suitable.

Therefore, we believe our inclusion of these AIGC-relevant forensics methods—particularly those utilizing frequency priors—is more aligned with the nature of our benchmark. That said, we acknowledge that our current baselines might appear limited, and we are open to incorporating more recent models in future versions of the paper. We sincerely welcome further discussion with the reviewer to reach a consensus.

Response to Weakness 4: Robustness Against Perturbations

We agree that robustness toward different types of perturbations is a crucial aspect of DeepFake detection. To investigate this, we conducted robustness experiments using five commonly used perturbation types:

• GB: Gaussian Blur
• BW: Block-wise Masking
• CC: Color Contrast
• CS: Color Saturation
• JC: JPEG Compression

We selected representative detectors from three categories, including our proposed VLForgery. For simplicity and to reduce resource cost, we applied a single fixed severity level for each perturbation type. The detailed results are shown in the table below.

Key observations include:

• JPEG compression (JC) tends to significantly affect most detectors.
• There is noticeable variation in robustness across detectors and test sets.
• Our method demonstrates a relatively balanced robustness profile — unlike SRM, which fails entirely under JC, or F3Net, which collapses in performance on fully synthesized samples.

I’d like to share some of our thoughts here in hopes of continuing this discussion and eventually reaching a common understanding.

Response to Weakness 1 and Question 1: Limited Novelty and Differences from Existing CoT/Few-shot Methods

We humbly disagree with the claim that our work simply combines existing components into a heavily engineered system.Instead, we propose a novel framework that fine-tunes the MLLM by injecting EkCot knowledge, enabling it to perform detection, localization, and attribution of AI-generated forgeries in a unified manner, rather than relying on auxiliary classifiers or handcrafted prompts.

To compensate for the limitations of existing datasets, we curated a new dataset from scratch, focusing on a diverse range of forgery types. Importantly, GPT-4o was used solely as an auxiliary tool during the dataset creation process, helping to generate descriptions. It was not involved at all in our model architecture, inference pipeline, or explanation generation.

As for EkCot, its motivation and mechanism are quite different from previous chain-of-thought or few-shot prompting strategies:

1. Many previous works [1-5] in explainable AIGC forensics suffer from either (1) subjective, ambiguous annotations (often generated by ChatGPT or human annotators), or (2) many works use ChatGPT to generate explanatory captions, and then fine-tune another model (e.g., LLaVA or LLaMA) with them. However, these models may have different internal representations and biases, and forcing one model to learn another model’s reasoning maybe harms the forensic performance. .

• [1] DDVQA
• [2] FFAA
• [3] x2-DFD
• [4] FakeShield
• [5] ForgerySleuth

2.These methods typically input only a single image when generating explanations. This is problematic because：

• Most open-source MLLMs are primarily tuned for high-level semantic understanding rather than forensic reasoning.
• AIGC images today are of extremely high visual quality, making superficial or semantic-level explanations uninformative.
• More importantly, relying on a single image forces the model to hallucinate plausible explanations without a grounded basis, since current MLLMs lack inherent forensic capabilities. In such cases, the model may fabricate reasons for why an image is fake, especially when no real reference is provided.

Our Approach

To address this — how to generate trustworthy explanations.

Our key insight: If we want the MLLM to generate explanatory information that is both beneficial for the model's own training optimization and maintains a certain degree of credibility, we must respect the model’s own judgment bias — i.e., what it believes are the most distinguishing features between real and fake images.

So, we designed the Description Generation Module (Sec. 3.2) with the following principles:

• We use the same MLLM for explanation generation and fine-tuning.
• Instead of using a single image (which leads to randomness), we input a pair of real and fake images, so the model can compare and find meaningful differences.
• We aggregate the outputs across multiple pairs to extract dominant low-level differences, which we define as the model’s judgment bias.

This avoids subjective labels, respects model reasoning, and grounds explanations in consistent behavior — distinguishing our method from prior work.

Response to Weakness 2: Lack of Insights and New Findings

We would like to clarify that several insights and novel contributions are discussed throughout our other responses:

1. A newly constructed explainable AIGC forensic dataset

   • Face-focused.

   • Contains partial and fully synthetic images

   • It contains long prompts and template-based facial prompts, designed to generate diverse and realistic content.

   • It is annotated with explanation-level labels.

2. A unique strategy to extract judgment bias from MLLMs
   We propose a Low-Level Feature Comparison Pipeline to mine the model’s internal judgment cues. This aligns the explanation process with what the model itself perceives, rather than relying on external interpretation.

3. Exploration of MLLMs' ability to localize manipulated regions through language
   We provide qualitative and quantitative analyses showing that MLLMs can not only make binary real/fake decisions but also generate fine-grained textual descriptions pointing to manipulated areas. This opens a new direction for text-based localization in multimodal forensics.

Response to Weakness 3 :Reliance on Closed-Source Models and Substantial prompt engineering.

In our framework, GPT-4o is not involved in the model’s training or inference pipeline.

As for the concern regarding prompt engineering: it’s important to understand that prompt design is a central challenge in explainable multimodal forensics. Generating reliable and faithful explanations from MLLMs inherently requires thoughtful prompt strategies.

In fact, all recent works [1-6] in this area rely heavily on handcrafted or dataset-specific prompts. For example, [6] introduces an additional classifier to improve detection performance, yet still falls back on traditional prompt-based explanations without fundamental changes. Without meaningful innovation in prompt formulation or explanation mechanisms, the field of MLLM-based forensics cannot truly progress.

• [6] KFD

Response to Weakness 4: Comment on Writing Quality, Terminology, and Figures

Thank you for your feedback. We will revise the paper to explicitly define all previously undefined technical terms and abbreviations.

Response to Question2: Use of GPT in Multiple Steps (Lines 138, 201, 225)

Why Use GPT-4o in Dataset Construction?

In our dataset, some partially manipulated images are generated by modifying specific regions of real images. In such cases, the visual quality and structure of the base image are preserved, and prompting quality has limited impact on visual realism. Using GPT-4o to generate descriptive prompts in this step allows for:

• Increased diversity in the types of image modifications.
• Semantic richness, since GPT-4o can create more varied and natural attribute combinations.
• Better coverage of edge cases, making the dataset more comprehensive and robust.

Why Use Manually Designed Prompts for Fully Synthetic Images?

In contrast, fully synthetic images (especially human faces) are extremely sensitive to prompt design. Randomly generated prompts from GPT-4o often lead to unstable or low-quality results due to lack of control over critical attributes.

Therefore, we chose to use hand-crafted prompts specifically tailored to generate high-quality synthetic human faces. These prompts were inspired by real-world image generation use cases (popular prompt styles from YouTube creator communities).
We designed our prompts carefully to ensure high-quality and diverse image outputs, as supported by visual results in our paper.

Comparison with Other LLMs and Manual Prompts

While we used GPT-4o in our implementation, we would like to emphasize that other LLMs (e.g., GPT-3.5, Claude, Gemini, etc.) could also be used for generating image-related prompts, as they all have access to rich linguistic and world knowledge. We simply chose GPT-4o based on convenience.

Response to Question3:Use of GPT-4o on Line 201, 225

Motivation for Using a Range of Question Formats

Inspired by [7] — which showed that different prompt wordings can lead to noticeable performance differences when using LLMs (e.g., ChatGPT) for real vs. fake image detection. Inspired by this, we followed a similar strategy and used ChatGPT-4o to generate multiple semantically consistent but syntactically diverse question formats.

• [7] Can ChatGPT Detect DeepFakes?

Our goal was to improve the overall robustness and generalization of the MLLM's forensic reasoning ability.

However, in our experiments, we found that while using a range of formats does bring slight variation, the impact of question phrasing was relatively minor, especially when compared to other components in the pipeline. Even using a single unified question led to comparable detection performance. Therefore, we decided not to emphasize this part in the main text.

Response to Question4: Undefined Technical Terms

Judgment Bias and Judgment Bias Description (Lines 178–183)

• Judgment Bias refers to the model’s internal reasoning cu* when determining whether an image is real or fake.

  For example, if we ask ChatGPT: “This image is fake — can you tell me why?” and it responds: “Because it looks blurry,”

  then the keyword "blurry" is ChatGPT’s judgment bias.

• Judgment Bias Description refers to the generated explanatory statement that incorporates one or more judgment biases in natural language form.

  That is, the judgment bias is only the core evidence (e.g., "blurry"), but the description is the full sentence expressing it in an interpretable way.

We also want to clarify a writing ambiguity in our current draft:

The "specific template" mentioned in Line 183 is the same as the "general rule template" mentioned in Line 189.

Acronyms in Table 6 (e.g., "FS_SDXL", "PS_Kan2.2")

These acronyms follow the same naming rule as described in the caption of Table 4:

• FS_SDXL - fully synthetic images generated by the Stable Diffusion XL (SDXL) model.
• PS_Kan2.2 - partially manipulated images generated using the Kandinsky 2.2 model.

Response to Question 5: Clarity Issues in Figure 2 and Figure 3

Unfortunately, due to NeurIPS rebuttal policy restrictions, we are unable to include revised figures or provide external links at this stage. We appreciate your understanding. We will update these figures in a future version of the paper.

We hope this message finds you well. We would like to gently follow up on our previous response to your review. If there are any remaining questions or if any part of our rebuttal is unclear, we would be more than happy to provide further clarification.

Thank you again for your time and consideration. We truly appreciate your efforts.

Response to Weaknesses 1: Generalization Issues Across Fully Synthesized Subtypes (e.g., InstantID)

We also noticed that performance consistently drops when InstantID is used as the test generator, regardless of the training source.

Upon closer examination, we believe a major reason lies in the unique nature of InstantID. As an identity-preserving generation method, InstantID retains facial identity characteristics from the reference (pristine) image, resulting in a feature distribution that closely resembles that of pristine images. This makes it inherently more difficult to distinguish from authentic images.

Furthermore, we observed that some training generators, such as Kandinsky, appear more capable of capturing and generalizing to these identity-preserving characteristics, while others like SD3 and Flux are more prone to confusion when encountering InstantID outputs.

Response to Weakness 2: Low Accuracy for Small Regions

To investigate why accuracy remains low for small regions (e.g., ears), we conducted a detailed analysis on the partially manipulated samples in our dataset. As shown in Table, we calculated the average proportion of each manipulated region in both the training and testing sets. The distributions are nearly identical, indicating no significant bias between them. For example, hair has the largest average proportion (34.50%), while eyebrows are the least represented (0.86%).

We further compared a variety of state-of-the-art manipulation localization methods and their performances across different regions, especially focusing on smaller areas like brow. Here are selected results for hair/brow ACC values:

From these results, we observe that models integrating frequency-domain knowledge (e.g., F3Net, SRM) tend to achieve better generalization to small manipulated regions. This suggests that augmenting MLLM-based explanation models with frequency-aware reasoning may help compensate for their current shortcomings in fine-grained localization.

We believe this is a promising direction, and we will continue to explore it in future work.

Response to Question 1: Robustness to MLLM Hallucinations and Effectiveness of EkCot

Our Core Insight

If we want the MLLM to produce explanations that are both credible and beneficial for model optimization, we must first respect the model’s own judgment bias — i.e., what the MLLM internally considers to be the most distinguishing features between real and fake images. This insight directly aligns with our objective (Line 165) of seek forensic features that the model itself can understand to discriminate the authenticity of an image, instead of imposing externally defined features.

How EkCot Addresses Hallucination

To tackle hallucination and guide the model toward more grounded reasoning, we designed the Description Generation Module (Section 3.2) based on the following principles:

• We use the same MLLM for both explanation generation and fine-tuning, ensuring consistency in the model's knowledge and representation space.
• Instead of using a single image (which leads to randomness), we input a pair of real and fake images, so the model can compare and find meaningful differences.
• We aggregate explanations across multiple image pairs to extract recurring, low-level forensic attributes — which we define as the model’s judgment bias.

This approach avoids handcrafted or subjective features, encourages the model to stay within its own representational bounds, and grounds explanation in behaviorally consistent visual evidence. These qualities distinguish EkCot from prior approaches that rely on static or external labels.

Empirical Evidence of Robustness

To evaluate whether the extracted judgment bias is effective — especially when facing subtle forgery traces like in partially manipulated images — we conducted ablation studies. We isolated the judgment bias from the EkCot pipeline and tested its impact on various subsets.

Table: Ablation Study of the VLForgery's forgery description generation component in the detection task. 'VF' , 'EKCot' and 'Jub' represent visual input features in forgery description generation, extrinsic knowledge-guided chain-of-thought, and Judgment Bias, respectively.

The results (provided in above Table) show that incorporating judgment bias improves detection performance on partially synthesized images, which are known to exhibit more subtle manipulations.

We appreciate the reviewer’s suggestion and will include additional results and analyses to further highlight these findings in the final version.

Response to Question 2: Lower Attribution Accuracy in Full Synthesis and Partial Synthesis Methods (Table 6)

1. Architectural Differences: Convolutional Models vs. Vision Foundation Models (VFMS)

In our study, traditional models, which are based on conventional convolutional architectures, tend to show very high attribution accuracy (around 98–99%) for Full Synthesis (FS) samples. However, their performance significantly drops when dealing with Partial Synthesis (PS) samples, with accuracy as low as 6%. In contrast, our method, which is built on vision foundation models (VFMS), exhibits more balanced performance across both FS and PS categories.

2. Explaining the Drop in Accuracy

We believe this drop in accuracy can be interpreted with insights from the paper [1], which discusses the asymmetric detection phenomenon in AIGC forensics. Traditional convolutional models are prone to overfitting to specific forgery patterns in the training data, especially in the FS case, where large-scale synthetic artifacts are easier to capture. However, these models struggle with generalization when encountering subtle or unseen manipulations (e.g., PS samples such as PS_SDXL).

• [1] Effort: Efficient Orthogonal Modeling for Generalizable AI-Generated Image Detection

In contrast, VFMS-based methods can better incorporate semantic-level information (e.g., scene context, object structure, identity consistency), enabling the model to form a higher-dimensional decision space that fuses both semantic content and forgery cues. This reduces overreliance on any single artifact or forgery pattern and leads to more robust performance across forgery types.

Response to Question 3: Robustness Against Unseen Perturbations

Due to the NeurIPS 2025 rebuttal policy that limits the response to 10,000 characters, we have addressed this concern under our reply to Reviewer fi2j's Weakness 4, as the two questions are closely related in nature. We sincerely appreciate your understanding.

Thank you again for your insightful comments. We would like to kindly follow up on our rebuttal to see if there are any remaining questions or clarifications needed. We are more than happy to discuss further if anything remains unclear.

We truly appreciate your time and consideration.

Thank you for your recognition of our work and the paper’s formatting quality. We appreciate your positive feedback.

Response to Weakness 1: Generalization to Unseen Generators and Real-World Forgeries

We appreciate the reviewer’s observation. Indeed, our motivation in constructing the VLF dataset was driven by the limitations we observed in existing face forgery datasets at the time, particularly with respect to partial synthesis using diffusion models. Most available datasets had limited partial forgeries and were often of low visual quality, largely due to the lack of carefully crafted prompts tailored to face synthesis.

For instance, we examined the dataset from [1] and found that many generated faces exhibited significant distortion and artifacts. In response, we developed a set of long, diverse, and high-quality prompt templates—partially inspired by observations from a YouTube channel (unfortunately, we are unable to disclose this due to policy restrictions). As shown in Section B.4, our generated face images achieve notably higher visual fidelity.

• [1] Diffusion Facial Forgery Detection

We acknowledge that generalization to unseen generators and real-world forgeriesis an important open direction. Given the emergence of newer datasets and generators in recent months, we plan to conduct broader generalization studies in future work. However, due to limited computational resources and time constraints, a full-scale evaluation on all unseen generators and real-world samples could not be completed for this submission.

Response to Weakness 2: Subjectivity in Handcrafted Low-Level Vision Biases

We acknowledge that this approach inevitably introduces a degree of subjectivity, especially in the filtering and bias selection process.

Inspired by prior work [2], which explores specialized models for low-level vision comparison (e.g., image quality assessment or structural similarity), we believe a promising future direction would be to incorporate multi-expert models or objective evaluation networks to generate more reliable and less subjective low-level vision biases.

• [2] Towards Open-ended Visual Quality Comparison

We recognize that this would be a significantly more complex engineering effort, but we see it as a valuable path forward, and we are committed to further exploring this direction in future work.

Response to Weakness 3: Low Localization Accuracy for Small Regions

To investigate why localization accuracy remains low for small regions (e.g., ears), we conducted a detailed analysis on the partially manipulated samples in our dataset. As shown in Figure, we calculated the average proportion of each manipulated region in both the training and testing sets. The distributions are nearly identical, indicating no significant bias between them. For example, hair has the largest average proportion (34.50%), while eyebrows are the least represented (0.86%).

We further compared a variety of state-of-the-art manipulation localization methods and their performances across different regions, especially focusing on smaller areas like brow. Here are selected results for hair/brow ACC values:

From these results, we observe that models integrating frequency-domain knowledge (e.g., F3Net, SRM) tend to achieve better generalization to small manipulated regions. This suggests that augmenting MLLM-based explanation models with frequency-aware reasoning may help compensate for their current shortcomings in fine-grained localization.

We believe this is a promising direction, and we will continue to explore it in future work.

Response to Weakness 4 and Question 1: Lack of Evaluation on Traditional Deepfakes

We acknowledge that the current version of our framework has not extensively evaluated performance on traditional Deepfake datasets, which may limit the generalizability of our conclusions.

Due to limited time and computational resources, we were unable to construct a separate training and testing pipeline on large-scale Deepfake datasets. However, as a preliminary effort, we evaluated our model on the face-swapping subset of FaceForensics++ .

Table: Evaluation on Traditional Face-swapping Deepfakes type. Results on different training and testing subsets using VLForgery. Accuracy is used for evaluation.

We agree that a more comprehensive study involving diverse Deepfake sources (e.g., face reenactment, lip-syncing, expression manipulation) would be valuable.

Response to Weakness 5: Computational Cost of Large MLLMs

Thank you for pointing this out — we acknowledge that we did not provide sufficient details regarding the computational cost.

In our current setup, training the model for one epoch using 8× NVIDIA RTX L40 GPUs takes approximately 6–8 hours. During inference, generating 100 tokens for a single image takes about 3–5 seconds on a single RTX L40.

Response to Question 2: Attribution Accuracy Gap Between Partial and Full Synthesis

We believe this issue is closely related to Weakness 3, which concerns low localization accuracy for small modified regions.

Our hypothesis is that attribution fundamentally relies on detecting consistent semantic patterns across an image. In the case of full synthesis, all regions of the image are generated by the same model and thus share a unified generative signature or semantic paradigm. However, for partial synthesis, only a small localized region follows the generative model's semantic pattern, while the rest remains real. This heterogeneity makes the attribution task more challenging.

As mentioned in our response to Weakness 3, incorporating additional frequency-aware features may help enhance the model’s ability to detect and attribute subtle manipulations. These features have shown promise in improving generalization to small forgery regions.

We also acknowledge the reviewer's point regarding possible dataset imbalance and insufficient signal. While we have balanced the number of examples per generator type, there is an intrinsic imbalance in region size and semantic coverage between full and partial synthesis samples, which could degrade attribution performance.

Since few prior works directly study attribution for partial synthesis, our current findings represent a preliminary step.

We appreciate the reviewer’s valuable suggestions and will consider these directions in future work.

I appreciate the authors' thoughtful rebuttal and the additional experiments provided. These responses satisfactorily address the concerns I raised in my initial review. After considering the reviews and the authors' clarifications, my decision is still Accept. Thank you for your efforts.