Guard Me If You Know Me: Protecting Specific Face-Identity from Deepfakes

We sincerely thank the reviewer for the positive evaluation of our work, including its novelty, technical soundness, strong experimental results, and the contribution of the VIPBench dataset. Below, we provide detailed responses to address the concerns raised and clarify any remaining questions.

---

Q1:

Which face-recognition model(s) did the authors actually use to compute similarity, and why were they chosen?

R1:

Thank you for your insightful comment. Table D1 reports the comparative results across the tested face-recognition models. Our ablation study shows that TransFace gives the best performance, so we used it empirically. We will add this explanation to the revised manuscript.

TABLE D1: Performance (AUC %) of VIPGuard using different face models as components.

---

Q2:

What is truly novel about casting deepfake detection as a fine-grained face-recognition task, and how does this differ from earlier identity-aware methods such as ICT-Ref?

R2:

Thank you very much for your insightful comment. Our method differs from previous identity-aware detection approaches in the following key aspects:

1. User-specific identity modeling: Our method uniquely models the protected user’s face by directly leveraging full facial information for detection. In contrast, prior methods focus on inconsistencies introduced by face-swapping operations rather than on the specific identity itself. For example, ICT-Ref detects discrepancies between inner and outer face identity embeddings, while Diff-ID compares differences before and after self-swapping. However, fully generative forgery methods often preserve identity consistency, causing these inconsistency-based methods to degrade significantly. As shown in Table 2 of our paper, our method maintains strong performance even under such conditions, highlighting the strong generalization ability of our method across different forgery types.

2. Use of local facial attributes: Unlike previous approaches that treat the face as a whole and rely mostly on global features, our method explicitly learns and utilizes local facial attributes. This enables it to capture fine-grained details that forgery methods struggle to replicate, leading to improved detection performance through the combination of global and local cues.

3. Leveraging MLLMs for semantic reasoning: By leveraging MLLMs to model high-level semantic facial attributes, our method effectively captures identity-related features and benefits from the strong generalization ability of large models. Moreover, the textual description of facial attributes enhances our model’s interpretability, which previous methods do not achieve.

---

Q3:

Could the authors provide the compute-resource details for reproducibility?

R3:

Thank you for highlighting this point. In the revision, we will add a table describing our hardware (multi-GPU workstation, multi-core CPU, and system memory) and software environment (recent Linux distribution, mainstream deep-learning framework with the corresponding CUDA/cuDNN stack).

---

Q4:

The appendix still lacks error bars or standard deviations for the experiments.

R4:

Thank you for your comment. In the revision, we will append error bars or standard deviation values to experiments, and briefly describe the procedure used to compute them. This will make the reported results more transparent and statistically sound.

---

Q5:

The checklist lists safeguards as “N/A,” yet the appendix outlines explicit safeguards for VIPBench, creating a contradiction that needs clarification.

R5:

Thank you for catching this inconsistency. We will update the checklist to “Yes” and ensure its description matches the safeguards already detailed in the appendix in the revised manuscript.

Thank you for your valuable feedback. We appreciate your recognition of the novelty of our problem and the uniqueness of our approach to addressing it, as well as the strength of our method, and the contribution of our benchmark. Below, we respond to your comments point by point:

---

Q1:

1. This paper assumes knowing the forgery scheme that will be employed by adversaries, which is a strong assumption.
2. How does the proposed method perform against an unknown forgery method?

R1:

Thank you for your comment. We list our response as follows:

• We have already presented the experiments involving unknown forgery detection in our paper (generalization), with no leakage of identity information or prior knowledge of the forgery methods. Specifically, we have evaluated the performance of our method on the following 12 unknown forgery methods: BlendFace, Ghost3, HifiFace, InSwap, MobileSwap, UniFace, ConsistentID, PuLID, GPT-4o, Jimeng AI, Tongyi, and Kling AI. We just utilized the SimSwap and Arc2Face as the training fake images in our method.
• Based on the proposed method, we enable MLLM to learn facial priors and reason with fine-grained facial information. Tables 1–4 in our paper show that our method consistently achieves the best generalization performance across all evaluated forgery methods.

---

Q2:

The paper focuses on 22 unique identities in the dataset and evaluation.

1. How can your method be applied to a new identity beyond the 22 considered identities?
2. Do we need to retrain the whole model?
3. If the identity-specific characteristics are different, how can you make sure the learned detection scheme generalizes to unseen identities?

R2:

Thank you for your comment. We respond to your question as follows:

1. To apply VIPGuard to a new user, we need to train a user-specific VIP token $\mu$ using a few of his/her authentic images. We generate positive and negative face pairs from these images to train the VIP token $\mu$, while keeping the rest of the model frozen.
2. No, the whole model remains frozen. Only the lightweight VIP token (0.11M parameters) is trained for each new identity, highlighting excellent adaptability across different identities.
3. Our paper focuses on ID-aware deepfake detection, which is defined as detecting facial forgeries by effectively utilizing known identity information, aligning [1,2]. Therefore, handling unseen identity conflicts with this topic. For VIPGuard, at least one authentic image of the user is required for forgery detection. As the number of available images increases, the model’s detection performance improves (see Figure 1 in the appendix), demonstrating the scalability of our method.

[1] Diff-id, TDSC, 2024.

[2] ICT-Ref, CVPR, 2022.

---

Q3:

The proposed framework consists of several modules adapted from existing literature. It is unclear how novel the proposed method is. Please clarify your work's key technical contributions.

R3:

Thank you for your comment. We provide a detailed explanation to clarify our novelty as follows:

• Firstly, to better address your concern, we sincerely ask if you could provide more specific details about the mentioned existing literature, such as relevant references or method names.

• Secondly, we would like to re-emphasize that our work introduces novel contributions that have not been proposed in previous research. The key aspects of our novelty are as follows:

  1. Task-Specific Formulation:

     • We reformulate deepfake detection as a face comparison task guided by identity priors—a paradigm not explored in prior MLLM-based detection methods. This enables the model to detect subtle forgery artifacts by comparing a query image against the known facial characteristics of a specific individual ("VIP"). This shift in perspective is itself a conceptual innovation.

  2. Comprehensive and Custom Data Construction Pipeline:

     • Unlike existing methods that rely on off-the-shelf LLMs or generic annotations, we introduce a dedicated facial annotation model trained to extract precise identity, attribute, and similarity signals from faces. This allows us to generate high-quality, identity-centric training data that explicitly teaches the MLLM to reason about facial consistency—a critical capability missing in standard training setups.

  3. Progressive Learning Strategy:

     • We design a three-stage progressive learning framework—facial attribute learning → general face comparison → VIP-specific customization—that gradually equips the MLLM with the fine-grained facial understanding necessary for personalized detection. This structured curriculum addresses the MLLM’s initial lack of facial expertise and ensures effective adaptation to the target task.
     • As demonstrated in Table C3, naive end-to-end training (i.e., standard MLLM fine-tuning) performs significantly worse, confirming that our improvements are not due to scale or data alone, but to our tailored methodology.

• Finally, while our method is partially inspired by Cross Attention [3, 4] and Yo’LLaVA [5] in the design of certain modules, there is no overlap with these works in terms of target tasks or pipeline. We clarify as follows:

  1. Cross Attention [3, 4]: We use cross-attention to compare two facial images. However, this doesn’t lessen our method’s novelty, since cross-attention is just a standard deep-learning building block.
  2. Yo’LLaVA [5]: We propose the use of a VIP token to capture specific user identity information, which is used to discriminate between the subtle identity shifts, AI-generated or real. In contrast, Yo’LLaVA focuses on enhancing model performance for conversational tasks (such as certain personalized conversational tasks). To further clarify the distinction, we provide experimental results in Table C3, where Qwen-2.5-VL-7B serves as the baseline. The significant improvement we observe strongly verifies the novelty and effectiveness of our approach.

TABLE C3: A comparison (AUC (%)) between our method, naively training MLLM, and Yo’LLaVA. Our method shows the best results of detection.

[3] Transformer, NIPS 2017.

[4] CAT, ICME 2022.

[5] Yo'llava. NIPS 2024.

---

Q4:

Does our detector stay robust after typical image degradations—blur, Gaussian noise, and compression?

R4:

Thank you for the insightful comment. We have added a robustness study (Table C4-1) showing that VIPGuard remains highly effective under common degradations. ** Because the detector relies on high-level semantic features such as facial information, it is inherently less sensitive to these low-level distortions.** The degradation settings are listed in Table C4-2.

TABLE C4-1: VIPGuard's performance (AUC %) under image degradations. Higher levels indicate stronger degradation. The results represent the average performance of VIPGuard in detecting BlendFace, InSwap, Arc2Face, and PuLID.

TABLE C4-2: Degradation settings included: Gaussian noise ( $\mathcal{N}(0,\sigma^{2})$) applied in YCbCr space; Gaussian blur defined by kernel ($K$) and standard deviation ($\sigma$); and JPEG compression with quality factor (QF).

---

Q5:

1. Are there any failure modes you observed that are worth mentioning?

2. Please discuss potential adaptive forgery strategies that may render the proposed detection method less effective.

R5:

Thank you for your valuable comment. We provide a brief explanation below.

1. Significant variations in head pitch and yaw angles, such as noticeable rotation of more than 50°, affect facial features and identity information. However, this limitation is shared by all face-based methods, including face recognition systems.
2. Significant age differences between training and test data. For example, training is based on younger facial images, but testing involves older facial images. However, this issue, stemming from notable identity shifts, can be effectively mitigated by expanding the dataset.

Inspired by your comment, we plan to address these potential failure cases in future work by incorporating 3D facial information, learning identity-related temporal cues from videos, and exploring other complementary sources of information. We will include additional analysis related to these aspects in the revised version.

We sincerely appreciate your positive feedback on the novelty and logic of our method, the proposed benchmark, and the overall performance. Below, we provide detailed responses to the concerns you raised.

---

Q1:

Assessing VIPGuard without relying on textual annotations during training is helpful for understanding its real-world applicability.

R1:

Thank you for your valuable comment. As shown in the newly added Table B1, VIPGuard performs well in Stage 3 using only image inputs. Despite being trained without textual annotations, it shows only a slight performance drop, demonstrating its practicality for real-world use. We will include this experiment in the revised paper to improve it further.

TABLE B1: Performance (AUC(%)) of VIPGuard under different settings in the Stage 3. Images + Annotation indicates the joint use of both images and their corresponding textual annotations during training. Only Image refers to training with images alone, without accompanying annotations.

---

Q2:

There are some symbol errors in Eq. (5) and Eq. (6) of the paper, which should be fixed.

R2:

Thank you for your comment. We apologize for the errors in Eq. (5) and Eq. (6). They will be been corrected in the revised version as follows:

Eq. (5):

L(\theta)=-\sum^N_{i=1}log[p_\boldsymbol{\theta}(x_i|x_{i-1}, f_{ref}, f_{query},g)]

Eq. (6):

$L(\theta)=-\sum^N_{i=1}log[p_\theta(x_i|x_{i-1}, \boldsymbol{\mu}, f_{query},g)]$

Moreover, we provide the equation that describes the computation of the feature representation $g$ from the Cross-Attention module, as follows:

$g = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$

where $K$ and $V$ are the visual features of the reference image $I_{\text{ref}}$ in Eq. (5) or the VIP token $\mu$ in Eq. (6), after the linear projection layer; and $Q$ is the visual feature of the query image $I_{\text{query}}$, also after the linear projection layer.

---

Q3:

How does the number of available authentic images of each VIP user affect the final performance?

R3:

We appreciate your comment. We would like to clarify that this experiment was already included in Figure 1 of our appendix. In this experiment, using just three authentic images per VIP user leads to a significant improvement compared to the one-shot setting (i.e., one authentic image per user, equivalent to using only the Stage 1+2 model). Our results show a progressive performance gain as the number of available authentic images increases from 3 to 20. In conclusion, providing more authentic images per VIP user enhances the model’s ability to detect fake images.

---

Q4:

It is suggested that the authors detail the inference process of the method in the manuscript and also report the inference time.

R4:

Thank you for your comment. To perform detection, VIPGuard will use the following pre-defined template. In this template, FACE_SCORE is the facial similarity score and <|face_pad|> is the placeholder of the output of the cross attention module.

Please determine whether the person in the input image is VIP user. The face similarity of these two face is {FACE_SCORE}/100. The face tokens are shown as follows, <|face_pad|> . You should directly answer 'yes' or 'no' without any explanation.

For each VIP user, we input a set $\mathcal{X}$ of authentic images into the face model $E_{face}$ and calculate a centric vector $f_{center}$ by the following formulation.

$f_{center}=\frac{1}{N}\sum_{x\in{\mathcal{X}}} E_{face}(x)$

where $x$ is the input image and $N$ is the size of the set $\mathcal{X}$.

In inference, we will calculate the facial similarity score between the feature $f$ of the current input image $x$ and $f_{center}$, as shown following

$f = E_{face}(x)$

$sim = 100 \Biggl[ \frac{1}{2} \Bigl( \frac{f \cdot f_{\text{center}}} {\lVert f\rVert_{2} \cdot \lVert f_{\text{center}}\rVert_{2}} + 1 \Bigr) \Biggr]$

Finally, the input image, facial similarity score, VIP token, and textual template are fed into the MLLM to produce the final output. Moreover, when Qwen-2.5-VL-7B is selected as the backbone, the inference time is about 2.09 seconds per sample without any acceleration techniques.

---

Q5:

It is suggested that the authors place the problem formulation in the appendix to the main content, which could help readers better understand the specific setting of this work.

R5:

Thank you for your valuable suggestion. We will revise the manuscript accordingly.

We sincerely appreciate the reviewer's detailed feedback and are encouraged by the positive assessment of the SOTA detection performance and explainability of our method, and the novelty of the proposed personalized deepfake detection benchmark.

Below, we provide point-by-point responses to the concerns raised by the reviewer.

---

Q1:

VIPGuard follows standard training process of MLLMs, making the novelty limited.

R1:

We sincerely appreciate the reviewer’s feedback. However, we would like to clarify that, our approach is fundamentally not a naive application of standard MLLM training pipelines, but rather a purpose-built framework specifically designed for the unique challenges of our task, i.e., personalized deepfake detection.

While MLLMs provide a foundational architecture, directly applying standard MLLM training paradigms to this task leads to poor performance, as shown in our ablation studies (Table A1). This is because personalized deepfake detection requires fine-grained understanding of facial identity priors—such as subtle anatomical structures and local attributes—that vanilla MLLMs, trained on general vision-language tasks, inherently lack.

To overcome this limitation, we propose VIPGuard, a novel and comprehensive framework that goes far beyond standard MLLM training in both design and execution. Our key innovations include:

1. Task-Specific Formulation:

   • We reformulate deepfake detection as a face comparison task guided by identity priors—a paradigm not explored in prior MLLM-based detection methods. This enables the model to detect subtle forgery artifacts by comparing a query image against the known facial characteristics of a specific individual ("VIP"). This shift in perspective is itself a conceptual innovation.

2. Comprehensive and Custom Data Construction Pipeline:

   • Unlike existing methods that rely on off-the-shelf LLMs or generic annotations, we introduce a dedicated facial annotation model trained to extract precise identity, attribute, and similarity signals from faces. This allows us to generate high-quality, identity-centric training data that explicitly teaches the MLLM to reason about facial consistency—a critical capability missing in standard training setups.

3. Progressive Learning Strategy:

   • We design a three-stage progressive learning framework—facial attribute learning → general face comparison → VIP-specific customization—that gradually equips the MLLM with the fine-grained facial understanding necessary for personalized detection. This structured curriculum addresses the MLLM’s initial lack of facial expertise and ensures effective adaptation to the target task.
   • As demonstrated in Table A1, naive end-to-end training (i.e., standard MLLM fine-tuning) performs significantly worse, confirming that our improvements are not due to scale or data alone, but to our tailored methodology.

In summary, our work does NOT naively apply MLLMs to the task. Instead, we identify the limitations of standard MLLM training in the context of personalized forgery detection and introduce a series of task-driven innovations—from data construction to model training—to overcome them. The substantial performance gains in Table A1 and the ablation studies strongly support the necessity and effectiveness of our proposed framework.

TABLE A1: Comparison (AUC %) between VIPGuard and Naive MLLM Training across four forgery methods.

---

Q2:

It would be more compelling if the VIPGuard could be applied to other MLLMs, such as LLaMA-3.2-Vision, Qwen2.5-VL-3B, etc.

R2:

We appreciate your valuable comment. Following your suggestion, we conducted an additional ablation study of altering different MLLMs in the VIPGuard.

Table A2 shows that our method consistently achieves strong detection performance on personalized deepfake detection across various MLLMs. These results will be incorporated into the revised manuscript to strengthen its overall quality.

TABLE A2: Performance Comparison (AUC (%)) of Our Method with Different MLLMs.

---

Q3:

More comprehensive ablation studies would be beneficial. (for Stage 2, Stage 3, and Stage 2, 3)

R3:

Thank you for your valuable comment. We have added additional results in Table A3 to demonstrate the effectiveness of Stage 2, Stage 3, and Stages 2,3. Due to MLLMs’ limited understanding of faces, directly applying Stage 3 results in poor performance. In contrast, enhancing the model’s basic perception through Stage 2 significantly improves performance, further demonstrating the effectiveness of the proposed progressive framework. These results will be included in the revised version of the paper, and we believe they will strengthen our work.

Table A3. An ablation study on the performance (AUC (%)) of different components in VIPGuard

---

Q4:

The proposed method seems to be sensitive to the first image input. Would you suggest analysis about it?

R4:

Thank you for your comment. The first input image serves as the reference ($I_{ref}$ in Figure 4 of our paper), representing the facial image of the protected user. In our method, we replace it with a VIP token to mitigate the sensitivity issue.

• Challenges may arise when detection relies solely on a single reference image without training the VIP token $\mu$, a setting referred to as OneShot. Under this setting, changing conditions may alter the image attributes, making detection to variations in the initial input.
• In contrast, our method replaces the first image with a learned VIP token obtained through training. This token encodes the user’s identity from multiple images by training, resulting in a more robust and stable representation. Table A4 shows that VIPGuard consistently outperforms the OneShot setting, confirming its robustness and effectiveness.
• Figure 1 in the appendix shows that performance improves as more authentic images are encoded into VIP tokens, highlighting their effectiveness in mitigating sensitivity.

TABLE A4. Performance (AUC (%)) comparison between the OneShot and VIPGuard. VIPGuard, which utilizes a learned identity token trained from multiple images,

---

Q5:

VQA data for VIPBench, which is used to train VIPGuard, is generated by GPT-4o and Gemini 2.5 Pro.

1. If those models are already able to explain visual attributes well, why are their deepfake detection capabilities lower than VIPGuard's?
2. Is it possible to get better results from those models by prompt engineering?

R5:

1. The key difference lies in the information available to the model when functioning as a Captioner versus as a Detector. This distinction stems from the differing roles played by Gemini 2.5 Pro (and similar models) in each case:

   • Captioner: Generates textual annotations for MLLM training. During this process, we explicitly provide rich prior information (e.g., category labels), and the Captioner formats it into the desired textual structure.
   • Detector: Performs detection based on the input image and question, but is not allowed to access any prior information at inference time.

   During annotation generation, Gemini 2.5 Pro, acting as a Captioner, was given full prior information—including category labels, as shown in the following equation (also Eq. (2) in our paper):

   $Prompt = Gemini(Q, f(I_i, I_j), I_i, I_j, G(I_i), G(I_j))$

Here, $Q$, $f(I_i, I_j)$, and $G(I_i), G(I_j)$ denote priors such as category, facial similarity, and facial attributes. Gemini 2.5 Pro and other commercial models merely format this information into text; they do not perform any detection themselves.

2. These models cannot detect effectively during testing—when such priors are still unavailable—even with prompt engineering. In contrast, VIPGuard is trained to perceive this information and perform detection independently, without relying on external priors.

Thank you for your prompt reply. We appreciate your feedback and are pleased that our previous reply has addressed most of your concerns. We reply to your new concerns as follows:

---

Regarding the novelty of our methodology, we would like to clarify the following:

1. Standard MLLM training is insufficient:

   • Simply and directly following the standard MLLM training procedure—without any modifications or optimizations—results in very low performance (with the value 65.34% of AUC in Table 1), while our proposed training paradigm achieves 98.23% (+32.89% over the Standard MLLM training) in Table 1.
     • This significant performance improvement highlights the necessity of developing a new and more effective training strategy (not just directly following the standard strategy).

2. Key novel contributions of our approach:

   • Specifically, our approach introduces the following novel components that go beyond standard MLLM training practices:

     • We tackle personalized deepfake detection from a new perspective by reframing forgery detection as fine-grained facial identity recognition.

     • Our framework consists of three stages designed to train MLLMs to recognize subtle identity differences of the VIP users progressively, which is different from the standard training procedure.

     • To the best of our knowledge, this is the first method to introduce a face comparison mechanism within MLLMs, applied in both Stage 2 and Stage 3.

We hope that our latest responses have clearly addressed your questions and concerns. We truly appreciate your thoughtful feedback and the opportunity to improve our work. Thank you!

TABLE 1: Comparison (AUC %) between VIPGuard and Standard MLLM Training across four forgery methods.

---

Regarding the practicality of our method, we would like to clarify the following:

1. Even when multiple VIP tokens are present, our method can accurately verify image authenticity—crucially, it does so without any manual selection of VIP tokens. Specifically, we simply selected the VIP token according to the facial similarity score and call the method "Adaptive VIPGuard". The highly similar results of the original VIPGuard and Adaptive VIPGuard, as shown in Table 2, provide strong evidence of their effectiveness in practical scenarios. The details of the Adaptive VIPGuard are as follows:

   • VIPGuard first calculates the similarity between the input image and the held VIP identity, selecting the corresponding VIP token based on this similarity; the second stage applies the selected VIP token for detection, following the procedure outlined in the paper. No additional training steps or modules are introduced in this process.

2. Our method already meets real-world application needs. For high-profile individuals such as politicians and celebrities, the required detection accuracy is much higher, as even a single undetected counterfeit image can have serious societal consequences. Our method can achieve a very high detection performance to satisfy the demand.

3. Traditional deepfake detection methods fall short of this level of accuracy and currently offer no user-specific protection. Our method significantly outperforms existing approaches in terms of accuracy for VIP users, offering robust protection and effectively addressing a critical gap in the field.

TABLE 2: Comparison of AUC (%) for adaptive VIP token selection in VIPEval, where Adaptive VIPGuard refers to the automatic selection of the VIP token without manual intervention.