Debiasing Attention Mechanism in Transformer without Demographics

The authors would like to thank to the reviewer's for thorough and careful reviewing our work. We appreicate the insightful feedback and suggestions provided. To address the concerns raised, we offer the following clarifications:

Q1: Which method belongs to "Fairness without Demographics", which method requires Demographics?

Response: We note that for all the methods we compared, none of those utilize demographics in training procedures. We provide detailed illustration on implementation of all methods in Appendix Section C.

Q2: Comprision with SOTA methods.

Response: We recognize that our method does not outperform all other methods across every dataset. As shown in Table 2, Learning from Failure (LfF) attains better Equality of Opportunity (EOp) results, albeit at the cost of a roughly $6\%$ decrease in accuracy compared to the Empirical Risk Minimization (ERM) model. We also observe that the average accuracy of $87.97\%$ is only marginally higher than the class rate, where $p(y=0) = 84.62\%$ in the test set. However, our method achieves comparable utility to the ERM model while enhancing group fairness.

In Table 4, as we discussed in Section 4.2, the significant disparity between the rates $p(a=0)=92.9\%$ and $p(a=1)=7.1\%$ leads to an increased theoretical upper bound. As specified in Theorem 1, where $λ_0=p(a=0)$ and $λ_1=p(a=1)$, this upper bound is directly proportional to $\frac{λ_0}{λ_1}$. This disparity accounting for the performance drop in this test. Although Just Train Twice (JTT) shows superior fairness outcomes, it requires considerably more computational resources compared to other baseline methods.

Q3: Experiments on the energy controlled setting.

Response: In line with the reviewer's recommendation, we have conducted additional experiments with a specific focus on energy consumption. We set a predetermined energy budget and, within this constraint, trained various methods to observe their performance. The other experimental settings remain consistent with those detailed in Table 2 and Table 4 (in the paper). To ensure a fair comparison, we established the energy budget at the mean level of energy consumption observed across all methods. For the CelebA dataset, we allocate a power energy of 363.63 Wh, and for MultiNLI dataset, we allocate a power energy of 2742.78 Wh for BERT large, 867.46 Wh for BERT base. The results could be found in Table 1, 2, and 3.

Table 1: Results of the MultiNLI Experiment under a Limited Energy Budget of 2742.78 Wh (Backbone: BERT Base).

Table 2: Results of the MultiNLI Experiment under a Limited Energy Budget of 867.46 Wh (Backbone: BERT Base).

Table 3: Results of the CelebA Experiment under a Limited Energy Budget of 363.63 Wh.

Tables 1, 2, and 3 indicate that, under a limited energy budget, our method not only achieves improved utility but also yields enhanced fairness results simultaneously. In contrast, under the same energy constraints, the early stopping for JTT impedes its ability to achieve fair results. For LfF (Learning from Failure), the reduced number of iterations improves its utility. However, this comes at the cost of its fairness performance.

Q4: Hardware influence on Enery metrics.

Response: (1) We would like to clarify that in this study, we conducted controlled experiments, ensuring that all methods were executed under identical configurations. The number of Floating Point Operations (FLOPs) is used to quantify the total count of elementary machine operations. In line with the reviewer's recommendation, we employed FLOPs as the metric to evaluate all methods. We carry out the experiment on CelebA dataset, with $y=$Blond Hair and $a=$Male. We use the thop library to estimate the total training FLOPs.

Table 4: Comparative Analysis of FLOPs Required by Different Methods During the Training Stage

We are grateful for the reviewer's dedicated and thorough review of our work, as well as their constructive suggestions for its improvement. We would like to address the following concerns raised:

Q1: For NLP, the scheme of picking top value vectors may be less effective for long sequences. Dynamic selection based on attention may work better.

Response: We agree with the reviewer's opinion that dynamic selection could be an effective mechanism for long sequences. In our current method, we select the top $k$ vectors based on attention weights and concatenate these vectors. Subsequently, these concatenated vectors are transformed into the contrastive space using a nonlinear function $g(⋅)$, which is implemented via a Multilayer Perceptron (MLP). However, due to the constrain of the fixed length input of MLP models, incorporating dynamic selection (may vary the number of concatenated vectors) can be very challenging. We consider it a valuable area for future exploration.

Q2: The last layer retraining is convenient but provides no guarantees. Analyzing how bias propagates through the full network could further improve this.

Response: For the last layer retraining we integrate our debiasing encoder with a pre-trained model. During the training phase, we freeze the pre-trained model, focusing on training the debiasing encoder and classifier for the downstream task. As the pre-trained model remains frozen during training, biases inherent in its representations are unavoidable. Our proposed method functions as a debiasing encoder, designed to minimize sensitive information in the final representation. For details, we kindly direct the reviewer to our general response.

Q3: The contrastive loss operates locally on values. Extending the alignment more globally could potentially improve fairness further without sacrificing accuracy.

Response: Extending contrastive loss to include all patches is effectively the same as applying it to the representation $\mathbf{z}$. However, it's important to note that directly aligning the representation $\mathbf{z}$ does not necessarily contribute to fairness. Park et al [1] demonstrate that with the use of Supervised Contrastive Learning (SupCon) to pre-train a ResNet model, unfairness issue still exists. We attribute this phenomenon to the correlation between the target and sensitive labels. Since the network contrasts the entire representation, it can still leverage sensitive-related information.

In contrast, our proposed method involves two steps. The first is debiasing the attention weight, a crucial step as it plays a role in selecting high attention patches. The second step involves using screened local values for contrastive learning, focusing on target-related information, thereby enhancing fairness. Our ablation study, detailed in section F of the Appendix, reveals that employing local value alignment alone has a limited impact on promoting fairness.

Q4: Missing ablation study

Response: Kindly refer to our Appendix, we provide an ablation study in section F.

References:

• [1] Park, S., Lee, J., Lee, P., Hwang, S., Kim, D., & Byun, H. (2022). Fair contrastive learning for facial attribute classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10389-10398).

Q5: (1). Limited evalutaion. (2). Concern about task selection (3). Compatibility of existing work.

Response:

• Following the reviewer's suggestion, we have carried out additional experiments on the CelebA dataset, which is notable for its 40 attributes and widespread use in fairness research. Specifically, we focused on tasks commonly explored in fairness studies. For the CelebA dataset, the tasks are $y=$ Wavy Hair $a=$ male [1] and $y=$ Bags under eyes $a=$ Young [2]. The results can be found in Tables 3 and 4.
• For different datasets, we do not pick a particular task. Instead, we followed the experiments outlined in [3, 4, 5], which are widely recognized in fairness research.
• We provide an additional experiment to demonstrate the compatibility of our method. We conducted tests combining our approach with the Just Train Twice (JTT), the results are in Table 5. We thank the reviewer for the suggestion and will add full results to our final paper.

Table 3: Classification results on CelebA dataset: $a$= male, $y$ = Wavy hair.

*LfF makes all predictions to the same target group. We did a grid search on the learning rate from {$10^{-3}, 10^{-4}, 10^{-5}, 10^{-6}$} with optimizer {$Adam, AdamW$}.

Table 4: Classification results on CelebA dataset: $a$ = young, $y$ = Bags under eyes.

Table 5: Compatible experiment on CelebA: $a$ = Male, $y$ = Blond Hair.

Analysis of Tables 3 and 4 shows that our method outperforms other baseline models in various tasks. While Just Train Twice (JTT) maintains good fairness, it experiences a slight drop in accuracy. Interestingly, as Table 5 illustrates, combining our method with JTT leads to enhanced outcomes. This improvement is attributed to JTT's ability to modify the distribution among sensitive groups, thereby balancing the data distribution across different groups and potentially reducing the unfairness upper bound as outlined in Theorem 1. Moreover, our method's use of contrastive learning techniques further enhances the utility of the JTT approach.

References:

• [1] Han, Xiaotian, et al. "FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods." arXiv preprint arXiv:2306.09468 (2023).
• [2] Park, Sungho, et al. "Fair contrastive learning for facial attribute classification." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
• [3] Liu, Evan Z., et al. "Just train twice: Improving group robustness without training group information." International Conference on Machine Learning. PMLR, 2021.
• [4] Hong, Youngkyu, and Eunho Yang. "Unbiased classification through bias-contrastive and bias-balanced learning." Advances in Neural Information Processing Systems 34 (2021): 26449-26461.
• [5] Baldini, Ioana, et al. "Your fairness may vary: Pretrained language model fairness in toxic text classification." arXiv preprint arXiv:2108.01250 (2021).

The authors would like to thank to reviewer sKv2 for your careful review and constructive suggestions. We would like to address the following questions:

Q1 & Q2: Justify debiasing only on the attention mechanism is powerful enough. Why the optimization objective in Eqn. (2) can be a good approximation to the fairness metrics?

Response: Please kindly refer to our general response Q1 for details.

Q3: The difference between the training set estimated statistics and test set statistics. Also concern about distribution shift.

Response: Following the reviewer's suggestion, we run the experiment on CelebA, with the task $y$=Blond hair, $a=$Male. We report $||\mathbb{E}[\mathbf{Q}]||_F$, $||\mathbb{E}[\mathbf{K}]||_F$, $||\sigma[\mathbf{Q}]||_F$ and $||\sigma[\mathbf{K}]||_F$ in the test set. We also report running mean, and running standard deviation estimated in the training set.

Table 2: Statistics in the test set and estimated statistics in the training set

where $eq$ = $||\mathbb{E}[\mathbf{Q}]||_F$,

$eq_r= ||\mathbf{e_{\text{running mean for query}}}||_F$,

$diff_{eq}$ = $||\mathbb{E}[\mathbf{Q}]-\mathbf{e}_{\text{running mean for query}}||_F$,

$ek = ||\mathbb{E}[\mathbf{K}]||_F$,

$ek_r$= $||\mathbf{e}_{\text{running mean for key}}||_F$,

$diff_{ek}$ = $||\mathbb{E}[\mathbf{K}]-\mathbf{e}_{\text{running mean for key}}||_F$

$sq = ||\sigma[\mathbf{Q}]||_F$,

$sq_r$= $||\mathbf{s}_{\text{running std for query}}||_F$,

$diff_{sq}$= $||\sigma[\mathbf{Q}]-\mathbf{s}_{\text{running std for query}}||_F$,

$sk$ = $||\sigma[\mathbf{K}]||_F$,

$sk_r$= $||\mathbf{s}_{\text{running std for key}}||_F$,

$diff_sk$ = $||\sigma[\mathbf{K}]-\mathbf{s}_{\text{running std for key}}||_F$.

It is important to highlight that in practice, we can only estimate the statistics of the test set based on the running statistics from the training set, a common approach also utilized in batch normalization and layer normalization. Table 3 is presented as a reference to evaluate the quality of this estimation. Our observations from Table 3 indicate that the distribution shift in the CelebA dataset, for the task $y=$Blond Hair, $a=$Male, is relatively minor. We acknowledge that significant deviations between the statistics estimated by the running mean and standard deviation and those in the test set will have an impact on our method's performance, potentially leading to a decreased utility. In fact, the distribution shift is also a challenging problem in the ML community.

Q4: Clarification is needed on the "last encoder training" section.

Response: We please reviewer kindly refer to our Q2 in the general response.

Q6: Do you have to assume sensitive groups are balanced within classes?

Response: We do not pre-suppose that sensitive groups are evenly distributed within classes. The reasons are as follows: (1) If we assume sensitive groups are balanced within classes, research by [6] has shown that group unfairness can be significantly mitigated through ERM training. (2) In cases where each class primarily consists of a single sensitive group, samples from minority groups tend to have higher losses. This occurs because these samples are contrasted with those from different sensitive groups. By optimizing this loss, our proposed framework effectively mitigates the disparities in representation among different sensitive groups. (3) However, in the extreme case where class distinctions rely only on sensitive information, our method may not be effective. In such instances, other comparative methods like JTT also fail and work similarly to ERM training.

Q7: I would love to see the distribution of attention weights before and after normalizing, as well as before and after absolute valuing (as well as an ablation study of the two steps).

Response: We provide a visual attention allocation comparison of our method and ERM in Appendix Section E. We will also provide more ablation results on the distribution of attention weight of the two steps in the revised version.

References:

• [1] Sudhakar, Sruthi, et al. "Mitigating bias in visual transformers via targeted alignment." arXiv preprint arXiv:2302.04358 (2023).
• [2] Qiang, Yao, et al. "Fairness-aware Vision Transformer via Debiased Self-Attention." arXiv preprint arXiv:2301.13803 (2023).
• [3] Lahoti, Preethi, et al. "Fairness without demographics through adversarially reweighted learning." Advances in neural information processing systems 33 (2020): 728-740.
• [4] Khosla, Prannay, et al. "Supervised contrastive learning." Advances in neural information processing systems 33 (2020): 18661-18673.
• [5] Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." International conference on machine learning. PMLR, 2020.
• [6] Idrissi, B. Y., Arjovsky, M., Pezeshki, M., & Lopez-Paz, D. (2022, June). Simple data balancing achieves competitive worst-group-accuracy. In Conference on Causal Learning and Reasoning (pp. 336-351). PMLR.
• [7] Chai, Junyi, Taeuk Jang, and Xiaoqian Wang. "Fairness without demographics through knowledge distillation." Advances in Neural Information Processing Systems 35 (2022): 19152-19164.
• [8] Nam, Junhyun, et al. "Learning from failure: De-biasing classifier from biased classifier." Advances in Neural Information Processing Systems 33 (2020): 20673-20684. -[9] Liu, Evan Z., et al. "Just train twice: Improving group robustness without training group information." International Conference on Machine Learning. PMLR, 2021. -[10] Hashimoto, Tatsunori, et al. "Fairness without demographics in repeated loss minimization." International Conference on Machine Learning. PMLR, 2018.

Thank you for carefully reviewing our work, also thank you for your valuable suggestions. We want to take this opportunity to address the following concerns:

Q1: Motivation: Why is it important to debias attention mechanisms? Why do we need a new method?

Response: The Transformer architecture is well-known for its effectiveness, but studies [1, 2] have noted that it is affected by statistical biases. For a detailed discussion on the significance of debiasing attention mechanisms, kindly please refer to the general response to Question 1.

Current methods addressing fairness issues without demographics can be broadly divided based on the use of auxiliary networks. Methods that do not require auxiliary networks include Distributionally Robust Optimization (DRO) [10] and Just Train Twice (JTT) [9], while those employing auxiliary networks including Adversarial Representation Learning (ARL) [3], Knowledge Distillation (KD) [7], and Learning from Failure (LfF) [8].

Our approach is specifically designed to address fairness without demographics and without the need for an auxiliary network, thereby reducing parameters for better efficiency. Regarding existing non-auxiliary network methods, DRO, as pointed out by [3], is likely to be affected by outliers, leading to a drop in performance. JTT, on the other hand, involves a costly double-training procedure, with the second phase incorporating a larger training dataset. In terms of the method for an auxiliary network, Knowledge Distillation (KD) utilizes a teacher model that is often larger than the student network performing the downstream task. Learning from Failure (LfF) alternates optimization between two networks of the same size, leading to parameter inefficiency. Although the auxiliary network in Adversarially Reweighted Learning (ARL) is smaller compared to the main network, its incorporation of adversarial training methods can introduce instability issues. In contrast, our approach obviates the need for such auxiliary networks, thereby enhancing parameter efficiency. Furthermore, it eliminates the necessity for multiple training phases, thus aligning with the demands of time efficiency.

Q2: The motivation behind Eqn (2). Does imposing constraints on attention limit the expressivity of the model?

Response: Please refer to our general response of Q1.

Q3: Unclear notation. The relationship between input and attention mechanism needs clarification. What is $\mathbf{q}_{cls}$?

Response: We appreciate the feedback from the reviewer. In our work, we denote matrices using bold uppercase letters and vectors with bold lowercase letters. For instance, the vector $\mathbf{q}$ represents a vector sliced from the matrix $\mathbf{Q}$.

We will include additional illustrations to clarify how this vector is inputted into the attention mechanism. Specifically, each input sample first passes through an Embedding layer. Then, it goes a transformation via three linear modules, which map it to $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$, respectively.

For $\mathbf{q}_{cls}$, it represents the first token in the sequence, which is utilized for downstream tasks.

Q4: Lack of a detailed model illustration and clearer explanation of the function $g$.

Response: We provide an algorithm for the entire model in the revised version. The function $g(\cdot)$ in our model represents a nonlinear projection head, a component utilized in contrastive learning [4,5]. We follow this protocol and employ $g(\cdot)$ to map the top attention vectors into the contrastive space. In our implementation, $g(\cdot)$ is a two-layer MLP with ReLU activation.

Q5: Lack of ablation study.

Response: We provide an ablation study in Appendix section F.