Fair Deepfake Detectors Can Generalize

We sincerely thank the reviewer for the recognition of our work. In the following, we address each of the concerns in detail.

Weakness 1 - Confounding Factors and Correction of Eq (2):

We are very grateful for the reviewer's insightful comments. We totally agree that in a more comprehensive causal graph, one could incorporate more confounding factors (e.g., scene context, image style). In our current work, we focus on DD and MC to demonstrate that, once these two confounders are controlled, fairness positively influences generalization. As demonstrated in Section 3.2, controlling these confounders enables an estimate of the fairness→generalization effect, which is significantly positive. Specifically, we observe an improvement in AUC (≈+2.35 points ↑) when moving from low fairness to high fairness, with statistical significance (p < 0.001). Section 4 further confirms that fairness interventions lead to consistent generalization gains under our DD and MC controls. We fully agree that Eq. (2) can be extended to handle more confounders. We would like to introduce a unified confounder set CF, and formulate an updated equation: P(A|do(F = f)) = ΣP(A|F=f, CF=cf)·P(CF=cf). We will clarify that the original Eq. (2) is simply its special case when CF = {DD, MC}.

We are very grateful for the reviewer’s insightful comment on the more diverse confounders, and we fully agree that this is both sensible and practically valuable.

Weakness 2 & Question b - The Clarification of DD:

We apolpgize for this confusion, and we would like to first detail the procedure of DD stratification, and then discuss the other properties of the data distribution related to Weakness 2.

“DD stratification” means dividing the dataset into demographic subgroups based on the sensitive attributes (e.g. gender and race) and treating each subgroup as a distinct stratum. In our implementation, we considered the intersection of Gender (Male, Female) and Race (White, Black, Asian), yielding 6 subgroups in total :

• Male-White, Male-Black, Male-Asian
• Female-White, Female-Black, Female-Asian

Each of these subgroups represents a distinct portion of the overall data distribution (DD). We condition on these subgroups when estimating the causal effect of fairness on generalization. In other words, during our causal analysis we calculate AUC within each stratum and then aggregate . By accounting for the data distribution this way, we “neutralize” its confounding influence – any observed improvement in generalization can be attributed to improved fairness rather than differences in demographic data balance.

(Related to Weakness 2:) We appreciate the reviewer’s suggestion to consider additional latent confounders beyond DD. In our formulation, DD is defined by demographic groupings. This choice follows from our motivation to estimate fairness-driven causal effects through demographic disparities, which are the most salient factors in fairness.

That said, we agree that other latent properties of the data – such as image background, clothing, or picture setting – can correlate with sensitive attributes and potentially act as confounders. While these broader factors are not explicitly represented in our causal DAG, we address their influence implicitly through two mechanisms: 1) Latent feature normalization within demographic groups, which aligns feature distributions across subgroups and thus neutralizes subgroup-specific visual patterns that co-occur with sensitive attributes (e.g., if a race group is predominantly photographed in a specific setting). 2) Feature alignment loss, which encourages latent features of samples belonging to different subgroups but sharing the same real/fake label to be close in the feature space, thereby mitigating the model’s dependence on spurious correlations such as background-related biases. These modules functionally mitigate the effect of style- or context-related biases (even if not explicitly modeled as DAG nodes).

We appreciate the suggestion to broaden the DD definition in the DAG. Incorporating richer confounder sets would allow more fine-grained causal modeling.

Weakness 3 - Clarification on do(F) Intervention:

We sincerely apologize for this confusion. Our do(F) intervention is intended to enhance the fairness of an initially low-fairness model, and establish a low-fairness (F=0) versus high-fairness (F=1) conditions. This is achieved by the resampling strategy, which is a widely adopted approach that has been used in prior fairness research [1] [2]. This resampling-based intervention effectively breaks the usual link between sensitive attributes and model predictions, approximating an idealized scenario of improved fairness. Specifically, in our implementation, the low-fairness setting is achieved through standard training (baseline). We then apply the resampling strategy to enhance fairness, thereby obtaining a high-fairness model. Quantitatively, this resampling reduced the Skew metric of Xception from 2.22 to 2.06 and that of EfficientNet from 2.01 to 1.91. The full set of numerical results from our causal analysis has been presented in the appendix to reinforce the robustness of our findings.

[1] Social debiasing for fair multi-modal llms.

[2] Enhancing Fairness through Reweighting: A Path to Attain the Sufficiency Rule

Small Notes:

We apologize for the confusion caused by these oversights. We will correct them accordingly. For instance, we will revise “We enforce” in line 200 to “We define” to better align with the subsequent introduction of L_attr.

Question a - Latent Normalization:

We sincerely appreciate the insightful feedback, and the reviewer’s understanding is absolutely correct. The latent feature normalization** (subgroup-wise feature normalization) in our method is performed as a one-time operation on the final representation, not at every hidden layer. This design keeps the approach simple and focuses the normalization where group-wise distributional shifts manifest most – in the final embeddings – without repeatedly normalizing at each network layer (which may unnecessarily constrain the model’s internal representations).

Question c - Hyperparameter and Fairness:

We have added the results between these two hyperparameters and skew in the table below. Overall, improvements in fairness are generally positively correlated with improvements in generalization. The best performance is achieved when λ_attr reaches 0.7 and λ_ortho reaches 0.2.

In conclusion, we sincerely appreciate the reviewer’s positive perspectives of our work and will provide additional clarifications to address the concerns.

We would like to express our sincere gratitude to the reviewer for the positive assessment of our work. Below, we address each of the concerns in detail.

Weakness 1 - Contributions & Fairness Objective:

We sincerely appreciate your insightful comments. We would like to first outline our contributions and then discuss the fairness objective.

(Contributions) Our contributions can be broadly divided into two parts. First, we employ causal graph to elucidate the relationship between fairness and generalization in deepfake detection. Building on this, we uncover the previously obscured potential for improving both of the two objectives. Second, we present DAID, a framework that leverages causal analysis to explicitly control confounding factors, thereby concurrently enhancing both the backbone model’s generalization and fairness. Empirical results demonstrate that our approach consistently outperforms baseline methods.

(Fairness objective) The proposed demographic-agnostic alignment loss L_attr is indeed related to the notion of individual fairness – ensuring that similar individuals receive similar outcomes. In our context, “similar individuals” are those sharing the same class label (real or fake) but differing in sensitive attributes (e.g., gender or race). By encouraging the latent representations of same-class samples to be close regardless of demographic group, the proposed alignment loss pushes the model towards treating these individuals equivalently. We will cite the related work, such as the FACE method [1], in Section 2 and add a discussion on clarifying how our loss extends the concept to deepfake detection.

Moreover, we will explicitly situate our approach within general fairness definitions. Our primary fairness goal is to reduce performance disparities across demographic subgroups – in other words, to achieve a form of group fairness. In terms of common definitions, this aligns most with pursuing equalized performance (e.g., smaller gaps in accuracy between groups), which is what the used Skew metric captures. By making feature embeddings invariant to sensitive attributes, our method mitigates the model’s reliance on those attributes, thus helping to satisfy criteria related to demographic parity (outcomes independent of demographics). In the revised paper, we will include a dedicated discussion of these fairness notions and how DAID’s objectives relate to them.

[1] Two Simple Ways to Learn Individual Fairness Metrics from Data.

Weakness 2 - Meaning of X and Y, and Clarification of “Observations”:

We are sorry for the confusion. X and Y are abstract variables (i.e., nodes) in the causal graph. In the context of our study, i.e., the relationship between fairness and generalization, the variables X and Y are in fact F (fairness) and A (generalization), respectively. In particular, as demonstrated in Section 3.2, we define X to quantify the level of fairness intervention (i.e., whether an intervention method is employed to enhance fairness), while Y measures the model’s generalization performance (measured by AUC, which ranges from 0 to 1).

In Sections 3.1 and 3.2, the term “observe” denotes examining how the generalization performance A varies across different levels of fairness intervention F, thereby establishing the interplay between fairness and generalization. The “observational data” refers to the dataset and evaluation metrics used in this analysis. In our experiments, we employ the DFDC dataset and report results in terms of our fairness metric (Skew) and generalization metric (AUC).

Weakness 3 - Interpretation of ACE:

We sincerely apologize for this confusion. The ACE refers to the weighted improvement in the AUC achieved by the high-fairness model over the baseline low-fairness model, where the high-fairness model is obtained by intervention. Specifically, on the DFDC dataset, we first record each model’s prediction scores and compute the AUC within each each demographic stratum (dd, mc). Aggregating these stratum-specific AUCs using empirical weights yields $A_{\text{low}}$ = 52.08%, and $A_{\text{high}}$ = 53.98%, implying a raw gain of 1.90 percentage points (The detailed results for each stratified group are presented in Table 4 of the appendix). To assess statistical reliability, we perform stratified bootstrap resampling (B=1000) on the DFDC dataset, and recompute the weighted AUC difference. The resulting mean effect is $\Delta$ = 0.0235 (i.e., 2.35 percentage points) with a 95% confidence interval of [0.0186,0.0280] and two-sided p<0.001. We will incorporate these details into Section 3.2 to clearly present the ACE definition, its computation, and its statistical significance.

Weakness 4 - Interpretation of Equation (10):

Our sincere apologies for the confusion. In Equation (10) of the manuscript, we define $\tilde{h} = U^{\top}\hat{h}$, where $\hat{h}$ is the normalized feature vector and $U$ is a trainable orthonormal projection matrix. This $\tilde{h}$ is indeed the feature representation used in the alignment loss $L{\text{attr}}$. In other words, our demographic-agnostic alignment loss $L{\text{attr}}$ is computed on the projected features $\tilde{h}i$ and $\tilde{h}j$ (for pairs of samples with the same class label but different demographics). In the current manuscript, we acknowledge that the use of $\tilde{h}$ and $\hat{h}$ has led to confusion. To resolve this, we will reorder Equation (10) and use $\tilde{h}$ in the $L{\text{attr}}$ to avoid any ambiguity.

We will also provide a clear description of all trainable parameters and variables in the loss function. In the revised version, we will list: $\theta$ (parameters of the backbone encoder), $\phi$ (parameters of the classifier), $U$ (parameters of the projection matrix in the feature aggregation module), along with definitions of $h$, $\hat{h}$, and $\tilde{h}$. The training objective can then be written explicitly as a function $L_{\text{total}}(\theta, \phi, U) = L_{\text{cls}}(\theta,\phi) + \lambda_{\text{attr}}L_{\text{attr}}(\theta, U) + \lambda_{\text{ortho}}L_{\text{ortho}}(U)$. We hope that can clarify the use of $\tilde{h}$ and loss function.

Weakness 5 - Additional Metrics and Discrepancies in AUC:

We totally agree that including these evaluation metrics can improve the clarity and completeness of the results. In the revised manuscript, we will incorporate additional metrics such as Accuracy, True Positive Rate (TPR), and False Positive Rate (FPR). One sample table is shown below:

Regarding the discrepancies in AUC scores, this is a result of different experimental settings. In our experiments (as stated in Section 4.1), all models are trained on FaceForensics++ (FF++) and then tested on unseen datasets (DFDC, DFD, Celeb-DF) to assess generalization across domains. This cross-dataset generalization scenario is a significantly more challenging setting, which naturally yields lower absolute AUC values. In contrast, the 94.88% AUC from [2] is achieved when training and testing on DFDC itself (within-dataset evaluation). The two numbers are not directly comparable due to this difference in protocol. We have reproduced** baseline methods in our cross-domain setup for a fair side-by-side comparison. We will explicitly clarify this in the revised manuscript to avoid confusion. For instance, we will add a note such as: The baseline results reported in our tables are obtained under cross-dataset evaluation protocol (train on FF++ and test on other sets).”

[2] Improving Fairness in Deepfake Detection

Finally, we once again extend our gratitude to the reviewer for meticulous review and recognition of our work.

We thank the reviewers’ thorough evaluation very much and apologize for any confusion resulting from our oversight. Please find below our detailed responses, which address each of the reviewers’ concerns.

W1 & Q1 & Q2 - Causal Relationship Establishment and Confounder-Reduced Feature Validation:

We sincerely appreciate these insightful comments and would like to address the concerns from three perspectives:

1. Causal relationship establishment. Traditionally, fairness and generalization are regarded as being in a trade-off relationship. However, our preliminary experiments reveal that, in certain cases, enhancing the fairness of deepfake detectors can improve cross-domain generalization. For instance, random partitioning of the training set into distributionally distinct subsets or enhancing the backbone with additional regularization techniques or data augmentation modules can yield superior performance compared to the baseline model. This intriguing observation lead us to hypothesize that some latent factors may obscure the causal link between fairness and generalization. Thanks to some causal analysis cases [1] [2], we find that the variability in model performance bears a striking resemblance to the influence of confounders. Therefore, we explore the interplay between fairness and generalization through the lens of causal graphs, and obtain empirical support for the notion that improving fairness can lead to enhanced generalization—provided that confounders are properly addressed (As discussed in Section 3.2).

   [1] Domain Generalization via Causal Adjustment for Cross-Domain Sentiment Analysis

   [2] Graph Out-of-Distribution Generalization via Causal Intervention

   The performance of The performance of causal relationships. As shown in Table 1 of Section 4, once confounding effects are mitigated (e.g., using the Data Rebalancing module to control DD and the Feature Aggregation module to control MC), our method achieves improvements in both generalization and fairness compared to all baselines, including conventional generalization detectors and fairness detectors.

2. Confounder-reduced features. The proposed DAID approach comprises two modules that can be incorporated into the backbone model to mitigate the influence of confounders on the extracted features.

   The effectiveness of the resulting confounder-reduced features can be evaluated from two perspectives:

   • The ablation results presented in Table 2 of the original manuscript demonstrate the effectiveness of our confounder-control strategy. Specifically, applying either the Data Rebalancing module to mitigate DD or the Feature Aggregation module to address MC individually yields notable improvements in both fairness and generalization. These findings validate that that each of our two modules effectively controls its respective confounder, and that combining them yields the best overall performance.
   • The heatmaps in Figure 5 of the original manuscript further illustrates how our method adjusts confounders. Specifically, attention maps of baseline models are biased. For instance, the baseline model focus on the lips when processing male subjects but on the upper faces for female subjects, implying reliance on gender‐related cues. In contrast, the DAID model exhibits a consistent attention distribution across all groups. This provides a validation of our proposed approach: After controlling for confounders, our method ensures that the model does not rely on sensitive-attribute proxies when making decisions. Instead, it attends to more generalizable, forgery-related features, thereby enhancing cross-domain performance.

We sincerely thank the reviewer once again for their insightful comments. We will highlight these valuable insights more in the revised manuscript.

W2 - Exclusion of the “Others” Category:

We sincerely apologize for any confusion caused by this experimental choice and would like to offer the following two-fold clarification:

1. Data Annotation and Rationale for omitting “Others.” We directly adopt the annotations from previously studies [1]. In the annotations, only three racial annotations—White, Black, and Asian—are explicitly defined. Images that do not fit these three groups are collectively labeled as “Others.” This “Others” group is both quantitatively minor (e.g, <5% in the FF++ dataset) and qualitatively heterogeneous (potentially comprising multiple smaller ethnic subgroups). Therefore, we restrict our analysis to the three racial groups to ensure that our evaluation is statistically sound.

2. Impact on experimental results. To ensure a fair comparison, we have reimplemented all baseline models. As illustrated in Figure 4 of the original manuscript, our method consistently achieves improvements across all subgroups.

   [1] Analyzing fairness in deepfake detection with massively annotated databases.

W3 & Q2 – Statistical Testing:

We sincerely appreciate the insightful comment. In the final manuscript, we will augment our results with multiple independent trials to assess stability. Specifically, we will train the primary experiments five times using different random seeds and report the mean ± standard deviation across these runs. A concrete illustration of this format is provided in the table below.

In summary, we sincerely thank the reviewer for these revision suggestions and apologize for the confusion we have caused. We will correct these issues and add new clarifications.

We would like to express our sincere gratitude to the reviewers for the valuable feedback on our work. Below, we address each of the concerns one by one.

Weakness 1 - Discussion on Fairness and Generalization May Be at Odds:

We sincerely appreciate this comment. Clarifying the inherent conflict between fairness and generalization is imperative.

When and why they are at odds. We totally agree that this conflict may arise from both data and model characteristics. Pertaining to the data aspect, the most representative factor, i.e., imbalanced distribution can lead to increased bias toward the majority group, thus improving generalization under certain limited datasets or scenarios. This however, poses the fairness problem a great challenge. Even worse, existing models tend to amplify this imbalance distribution problem, making the prediction biased. Though some efforts have tried to equalize performance across demographic groups, the improved fairness often comes with degraded generalization. Unlike the existing methods, in this work, we propose to leverage the causal theory with the confounder controlling guidance. Enlightened by this, our proposed method in fact aims to rebuild balance from imbalance. Therefore, we can maintain the generalization capability of vanilla models, and can also improve the prediction fairness.

When conflicts are difficult to resolve. The mitigation of such conflict also has its limitations. For instance, in certain highly specialized application scenarios, trade-offs between fairness and generalization may be inevitable. For instance, in high-security systems, developers might prioritize reducing overall overall misclassification, thereby sacrificing the performance on minority groups (e.g., individuals wearing unusual clothing). In contrast, for judicial models such as sentencing decisions or crime risk prediction systems, fairness across different demographic groups must be prioritized, even if it comes at the cost of reduced generalizability.

We greatly appreciate the reviewer’s insightful discussion on this issue. These discussions help us strenghthen the quality and rigor of this work. We will highlight these new points in the revised manuscript.

[1] Improving model fairness in image-based computer-aided diagnosis

[2] The cost of fairness in classification

Weakness 2 - Related Work:

Thank you very much for highlighting related work on joint fairness and robustness. We will cite such work, including the tabular-data methods [1], and expand Section 2 of the original manuscript to discuss them. These updated discussions are following:

several of these approaches, especially [1], use distributionally robust optimization to improve worst-group performance, thereby addressing fairness and robustness together. Unlike these fairness techniques that operate on tabular data, our DAID framework is tailored to visual deepfake data for robust cross-domain deepfake detection. Leveraging fairness as a causal intervention, DAID simultaneously boosts fairness and cross‐domain robustness (generalization).

[1] Fair-n: Fair and robust neural networks for structured data.

Weakness 3 - Minor Issues:

We sincerely apologize for these errors and will comprehensively revise all inaccuracies in the manuscript in accordance with the reviewer's comments. For instance, we will correct the typo “it worth noting” to “it is worth noting” on Line 273 . And we will remove venue names from Table 1 (and elsewhere) to avoid redundancy. We thank the reviewer for pointing them out.

Finally, we once again extend our sincere gratitude to the reviewer for the insightful comments.