FairLoRA: Targeted Bias Mitigation without Performance Loss

---

Response to Question 2:

Thanks for your question. In scenarios involving two sensitive attributes (Race and Sex), the worst-group is typically defined as the group with the lowest performance among all possible combinations of sensitive attributes and labels. In our experiments, this results in a total of 8 groups (2 Race attributes × 2 Sex attributes × 2 labels).

In multi-attribute scenarios, WGA primarily reflects the bias of the most underrepresented group, which often masks the nuanced effects of bias mitigation for individual attributes. For example, when mitigating gender bias, the WGA might still be dominated by the smallest group defined by race, thereby overshadowing FairLoRA's specific contributions to reducing gender bias.

This limitation makes WGA less effective at demonstrating the stepwise mitigation of biases across multiple sensitive attributes, which is a key focus of our framework. Thus, we did not include changes in WGA scores.

---

Response to Weaknesses 1:

We apologize for any lack of clarity in our initial presentation. In the revised version, we have reorganized the theoretical analysis section to better explain its purpose and conclusions.

The theoretical analysis aims to derive the conditions under which the FairLoRA framework can fine-tune a model without compromising the original model's performance. This analysis is specifically designed for the FairLoRA framework, where TPR (True Positive Rate) and FPR (False Positive Rate) represent the performance of the discriminator in FairLoRA, $M$ denotes the original model, and $M_{\text{lora}}$ represents the model after adding the LoRA module.

Our results show that when the discriminator in FairLoRA satisfies Eq.(7), the framework ensures that fine-tuning does not degrade the original model's performance. Moreover, we provide an approximate condition:$\frac{\text{TPR}}{\text{FPR}} \geq \frac{1-p}{p}$, where $p$ represents the proportion of sensitive labels. We also explain why this approximation is reasonable in most practical scenarios. This approximation offers a practical and interpretable way to assess the framework's ability to maintain performance during fine-tuning.

We hope this clarification addresses your concerns and demonstrates the theoretical grounding of our method.

---

Response to Weaknesses 2:

While prior works have explored fine-tuning using LoRA and adversarial learning with discriminators, our proposed framework differs significantly in its design and objectives. Specifically, our approach integrates a pre-trained discriminator capable of identifying sensitive attributes directly into the model architecture. Based on the discriminator's outputs, we selectively activate LoRA modules that are specifically trained to mitigate biases related to sensitive attributes.

The primary goal of our method is to enhance the model's fairness without compromising its original performance. To this end, we provide a theoretical guarantee tailored specifically to the FairLoRA framework. This theoretical guarantee ensures that models fine-tuned with FairLoRA can improve fairness under the condition of maintaining the original model's performance. Importantly, this guarantee is unique to the FairLoRA structure and does not generalize to other frameworks or architectures.

We believe this combination of selective LoRA activation and a performance-preserving theoretical foundation distinguishes FairLoRA from existing methods in the literature.

---

Response to Weaknesses 3:

Our method does require access to some labeled data. However, we believe that both labeled and unsupervised bias mitigation approaches have distinct advantages and application scenarios.

Labeled bias mitigation allows for precise identification of data or behaviors that contain biases, enabling a more targeted and accurate debiasing process while minimizing the risk of misclassification or omission. This level of precision is particularly valuable in scenarios where sensitive attributes are explicitly available or critical for the application context.

Since labeled and unsupervised bias mitigation methods serve different purposes and are suited to different domains, we argue that focusing on labeled bias mitigation should not be viewed as a weakness. Instead, it represents a deliberate design choice to address specific challenges that require a more controlled and interpretable debiasing process.

Response to Weaknesses 4:

Thank you for pointing this out. We apologize for the lack of clarity in our initial submission. We have revised the relevant sections to improve readability and ensure the training pipeline is easier to follow.

The details and algorithms were placed in the Appendix due to page limitations. However, in the revised version, we have restructured the content to provide better context and flow in Section 3, while keeping key explanations concise and accessible. Additionally, we have updated the caption of Figure 1 to provide a more comprehensive description.

We hope these revisions address your concerns and improve the overall clarity of the manuscript.

---

Response to Weaknesses 5:

Thank you for your valuable feedback. The fairness-accuracy trade-off is indeed a well-recognized phenomenon in this field. However, FairLoRA is specifically designed to improve fairness without sacrificing the model's original performance, distinguishing it from methods that rely on such trade-offs.

Because FairLoRA avoids trading accuracy for fairness, the fairness improvements may appear more limited compared to approaches that explicitly compromise performance. Nevertheless, our method ensures these improvements through both theoretical guarantees and experimental validation. From a theoretical perspective, we demonstrate that FairLoRA maintains accuracy while enhancing fairness under defined conditions. Experimentally, the results consistently show improvements in fairness metrics (e.g., WGA and EOD) while preserving accuracy.

Additionally, Experiment 2 (Table 2) demonstrates the incremental debiasing capability of FairLoRA. By storing information for different biases in separate modules, FairLoRA avoids catastrophic forgetting and achieves composability, enabling the effective mitigation of multiple biases over time.

While the fairness improvements may seem modest, they are significant in practical scenarios where maintaining performance is critical. We will further expand experiments in future work to comprehensively evaluate the method's potential across a broader range of datasets and metrics. We hope this explanation clarifies the novelty and effectiveness of FairLoRA in addressing the fairness-accuracy trade-off.

Response to Question 1:

Thank you for your thoughtful suggestion. We conducted additional experiments to investigate the impact of changing the debiasing sequence (e.g., gender → racial) and eliminating multiple biases simultaneously.

The results, as shown in Table 8, demonstrate that altering the order of debiasing has minimal effect on the final outcomes. This aligns with our explanation in the paper that FairLoRA possesses a "forgetting-avoidance" property, where corrections for different sensitive attributes are stored in separate LoRA modules, ensuring that adjustments to one attribute do not interfere with others.

Furthermore, Table 9 illustrates that whether biases are mitigated sequentially or simultaneously, the final results remain nearly identical. This is due to the fact that we store corrections for each bias in separate LoRA modules, which allows for independent adjustments to each sensitive attribute without negatively impacting the others.

Additionally, we focused on validating FairLoRA's ability to incrementally mitigate biases because, as society evolves, new biases will inevitably emerge. If each newly discovered bias requires retraining the entire model, it would result in significant costs and delays. Therefore, we believe that the ability to incrementally mitigate biases is essential for practical deployment in real-world scenarios.

Table 8: Performance and fairness comparison during progressive debiasing of sensitive attributes for DistilBERT-base and BERT-base

---

Table 9: Comparison of Progressive Debiasing and Simultaneous Debiasing Approaches

Response to Weaknesses 6:

Thank you for your insightful comment. We apologize if the presentation in Table 1 was unclear, leading to potential misunderstandings. Our FairLoRA Minority method is designed to fine-tune the model with the goal of improving fairness (measured by WGA and EOD) without sacrificing original performance (specifically, accuracy). In other words, we aim to enhance fairness while ensuring that ACC does not decrease.

In contrast, methods like ERM focus primarily on optimizing accuracy, which results in higher fairness metrics. When FairLoRA Minority is applied, ACC remains unchanged, while WGA and EOD show significant improvements. For instance, on the CelebA and MultiNLI datasets, WGA improved from 77.9 to 82.0 and from 67.3 to 71.0, respectively.

For fairness-oriented methods such as GroupDRO, DFR, and Lu et al., since these methods have already achieved relatively high fairness, the improvements from applying FairLoRA Minority are smaller. However, there is still a noticeable increase in fairness. These fairness-centric methods typically do not achieve the highest accuracy because they prioritize fairness over performance. On the other hand, applying FairLoRA Majority allows for further accuracy improvements without sacrificing fairness significantly.

Our method does not require a trade-off between fairness and performance, which is why the fairness gains are not as dramatic as those seen in methods that explicitly sacrifice performance to improve fairness.

---

Response to Question 1:

Responses to all weaknesses have been provided in the sections above.

---

Response to Question 2:

• Thank you for your question. We apologize for any potential confusion in the original manuscript. To clarify, in the experiments involving the same datasets, the models used for ERM, GroupDRO, DFR, and Lu et al.'s method are indeed the same. The differences among these methods lie in the fine-tuning strategies applied to the pre-trained models.

• Specifically, for the CelebA dataset, we utilized a 8-layer Vision Transformer (ViT) model. For the MultiNLI dataset, we employed the BERT-base model. These choices of datasets and models are consistent with those used in the methods we compared against in our experiments. In the revised manuscript, we have made further clarifications to ensure that the description is clear and avoids any ambiguity. We hope this resolves your concern, and we appreciate your attention to this detail.

---

Response to Question 3:

Thank you for your thoughtful question. The improvement in EoD(R) during the final stage of Experiment 2 can be attributed to the complex interplay between Race and Sex biases, which are not entirely independent of each other. These two sources of bias can interact in ways that amplify or mitigate each other's effects in the model.

For instance, certain data points may exhibit biased features related to both race and sex simultaneously. By mitigating one type of bias (e.g., gender bias), we may also reduce some of the associated bias in the other attribute (e.g., race bias). Specifically, certain race-gender combinations (e.g., a particular racial and gender group) may be subject to distinct forms of biased treatment. Eliminating gender bias could therefore help alleviate some of the racial bias present in these specific combinations.

We hope this insight helps clarify the observed improvement in EoD(R). We have made this reasoning more explicit in the revised manuscript for better clarity.

Response to Weaknesses 1&2:

We apologize for the lack of clarity in our original manuscript. We have revised the relevant sections to provide the requested details. Specifically, for Experiments 1 and 3, the model used for CelebA is a 8-layer Vision Transformer (ViT) model. For the MultiNLI dataset, we used the BERT-base model. These choices of datasets and models align with those used in the methods we compared against in our experiments.

Regarding the claim of ``state-of-the-art'' models, we acknowledge that our description may have been overly general. The methods we compared against are well-established and widely recognized in the field, but may not strictly qualify as "state-of-the-art'' in the current sense. We have revised the manuscript to remove this claim to ensure rigor.

---

Response to Weaknesses 3:

Thank you for your valuable feedback. We apologize for the lack of clarity regarding the dataset splits.

• For CelebA and MultiNLI, we used the official splits provided in the respective documentation, following the standard training and test set divisions.
• For HateXplain, since the official split is not provided, we followed the approach of Lu et al., where 50% of the samples were used as the test set.

Regarding the claim about the partial-information setting, we acknowledge that this description was not entirely accurate. We have revised this section to clarify that "FairLoRA operates under a partial-information setting, where group labels are observed only for a subset of the training set." Thank you again for pointing this out, and we hope the revision better reflects the experimental setup.

---

Response to Weaknesses 4:

Thank you for raising this point. Below, we clarify FairLoRA's design objectives and provide the rationale for our evaluation approach:

1. FairLoRA's Design Objectives and Unique Contribution
   FairLoRA is designed as a post-processing framework to enhance fairness while preserving the original model performance. Unlike methods that optimize fairness by sacrificing accuracy or other metrics, FairLoRA prioritizes a trade-off-free improvement, ensuring no degradation in performance. This fundamental design goal differentiates FairLoRA from other methods and positions it as a complementary, rather than competing, approach.

2. Rationale for Experimental Design

   • Compatibility and Generalizability: FairLoRA's strength lies in its compatibility with and enhancement of existing fairness methods, such as ERM, GroupDRO, and DFR. Therefore, instead of conducting one-to-one comparisons, we focused on demonstrating its synergistic effects when integrated with these methods. This approach highlights FairLoRA’s broad applicability and composability, which are central to its contribution.

   • Avoiding Misleading Comparisons: FairLoRA does not aim to maximize fairness improvement at any cost, as some methods designed for specific fairness metrics do. Direct one-to-one comparisons with such methods may misrepresent its value, as FairLoRA emphasizes balanced improvements across fairness and performance without sacrificing one for the other.

3. FairLoRA's Practical Value
   The experimental results are structured to emphasize FairLoRA's real-world practicality: its ability to enhance fairness across diverse methods without compromising baseline performance metrics.
   This trade-off-free and general-purpose nature addresses common limitations in fairness optimization and showcases its unique contribution.

We appreciate the suggestion and are open to incorporating additional head-to-head comparisons in future work to further contextualize its performance relative to other methods.

---

Response to Weaknesses 5:

Thank you for your valuable comments. The primary objective of our work is to validate the ability of FairLoRA to operate without requiring a trade-off between performance and fairness. As such, the specific choice of hyperparameters or models does not impact the core validation of our method.

However, we acknowledge the importance of providing full details for reproducibility. In the revised manuscript, we have added this information in the supplementary materials. Specifically, we report the random seeds used in our experiments, which are seed 5, 15, 25, 35, and 45. For all models, we used the default hyperparameter settings provided in the original research papers, as no further tuning was conducted.

We hope this clarifies the experimental setup, and we appreciate your suggestion for greater transparency.

Response to Weaknesses 1:

Thanks to the reviewer's insightful comments, we have made the following revisions and clarifications in response:

1. Regarding Thms 1 and 2:
   We fully agree with the reviewer's suggestion that Thms 1 and 2 are intermediate steps in the proof of Thm 3. In the revised manuscript, we have rephrased these as lemmas to more accurately reflect their role in the theoretical development.

2. Clarification of Theoretical Contribution:
   We acknowledge the reviewer's concern regarding the perceived obviousness of the conclusion in Thm 3, specifically the requirement for a high ratio between TPR and FPR. To avoid any ambiguity, we would like to clarify that the TPR and FPR referred to in Thm 3 are not those associated with the final classification results. Rather, they pertain to the metrics associated with our specifically designed discriminator (unique to the FairLoRA framework), which determines whether the LoRA module should be activated to adjust the model for improved fairness.

More importantly, the core contribution of Thm 3 is the demonstration of the specific conditions under which the FairLoRA framework can improve fairness during fine-tuning without sacrificing model performance. This result breaks away from the conventional trade-off between performance and fairness that is typically observed in existing methods. In other words, when certain conditions are met, the FairLoRA framework allows for a significant improvement in fairness without any degradation in original performance, thus overcoming the typical performance-fairness trade-off seen in prior work.

---

Response to Weaknesses 2:

We appreciate the reviewer's thoughtful question and offer the following clarifications:

1. Regarding the presence of sensitive attribute information in model representations:
   In most cases, the hidden layers of a pretrained model contain rich token representations that capture a variety of features, including information related to sensitive attributes. We aggregate these representations through an attention pooling mechanism and train a discriminator to classify whether sensitive attribute information is present. If such information is detected, the FairLoRA discriminator effectively activates the LoRA module to adjust the model and reduce bias.

2. If the hidden layers do not contain sensitive attribute information:
   In situations where the model has learned representations that are invariant to the sensitive attribute—either because the task itself does not depend on it or due to the use of adaptation techniques that explicitly encourage invariance to sensitive attributes—our discriminator will be unable to detect the presence of sensitive attribute information. In this case, we believe that no further intervention from the FairLoRA framework is necessary, as the model has already learned to be unbiased with respect to that attribute. Specifically, when the representations for sensitive and non-sensitive attributes are indistinguishable, no bias exists.

We appreciate the reviewer’s thoughtful comments and insightful suggestions, which have improved the quality of our work.

Response to W1:

Thank you for your review and comments. Regarding your concern about FairLoRA requiring balanced datasets, we provide the following clarifications:

FairLoRA's Objective:

FairLoRA aims to mitigate unfairness in machine learning models while maintaining or even slightly improving their original performance. A key innovation of our approach lies in the design of a discriminator, which effectively avoids the trade-off between performance and fairness—an issue that is prevalent in many existing fairness-enhancing methods.

Use of Balanced Datasets:

FairLoRA is a highly flexible framework that can be integrated with other fairness-enhancing techniques, such as GroupDRO, and does not rely on any specific dataset or method. In our study, we chose the simplest balanced dataset approach to demonstrate FairLoRA's capability to improve fairness without compromising performance. However, the framework can be extended to more complex settings using alternative methods.

Addressing Minority Samples:

FairLoRA leverages the representational capacity of pre-trained models, which allows it to remain effective even with limited samples from minority groups. By focusing on post-processing fine-tuning, FairLoRA ensures both fairness enhancement and performance preservation, even in data-scarce scenarios.

We hope these clarifications address your concerns, and we have also made adjustments to the manuscript to improve its descriptions and explanations for better reader comprehension.

---

Response to W2:

Thank you for your valuable suggestion. We have conducted additional experiments on CelebA using another combination of attributes to further validate FairLoRA. The results, as shown in Table 5, are consistent with our previous findings, demonstrating FairLoRA's robustness and effectiveness across different attribute combinations.

Table 5: Performance comparison across different attributes of CelebA dataset

---

Response to Comments:

The methods presented in [1] and [2] focus on reducing biases in pretrained models without compromising downstream task performance. These approaches are highly effective as upstream solutions but may face limitations when addressing biases introduced during fine-tuning on specific downstream tasks. In scenarios where the fine-tuning dataset itself is imbalanced, new biases can emerge that upstream methods cannot rectify. This is the key problem our research aims to address.

For instance, pretrained models fine-tuned on specific tasks such as classification in CelebA or MultiNLI often rely on datasets with inherent imbalances. In CelebA, attributes like “Blond hair” (14.91%) and “Heavy makeup” (14.37%) are underrepresented, which can lead to biases against these minority groups in the fine-tuned models. These biases may manifest as disproportionate errors for underrepresented groups, and such biases may be discovered over time as the model is deployed. In these cases, it becomes crucial to enhance fairness without re-training the model or sacrificing its original performance—a challenge our proposed method, FairLoRA, is specifically designed to address. To the best of our knowledge, FairLoRA is the first method that can effectively improve fairness in this context while preserving the original model's performance.

Additionally, thank you for pointing out the terminology issue with "African" and some other grammatical errors. We have made the necessary corrections. Thank you for your thoughtful suggestions!

Response to Q1:

Thank you for your thoughtful question regarding the loss function used for training the discriminator. Below is a concise explanation:

Loss Function Used:

We employ the GroupDRO loss function to train the discriminator, as it is specifically designed to address fairness-related tasks by minimizing disparities in classification performance across groups.

Framework Flexibility:

FairLoRA is not limited to GroupDRO and can integrate other loss functions, such as binary cross-entropy for standard tasks or advanced fairness-oriented losses tailored to specific objectives.

Core Philosophy:

FairLoRA enhances fairness through post-processing fine-tuning without compromising the model's original performance. The choice of loss function impacts fairness improvement but does not degrade model performance.

We have revised the manuscript to clarify the discriminator's training process and the role of the loss function. Your feedback has been invaluable in improving our work.

---

Response to Q2:

Thank you for pointing this out. You are correct that $M$ refers to the model. We will explicitly define this notation in the revised manuscript to ensure clarity and avoid any potential confusion for readers.

---

Response to Q3:

Thank you for your insightful comment regarding the dataset selection. Below is our detailed explanation:

Choice of Datasets (CelebA, MultiNLI, HateXplain):

We selected CelebA, MultiNLI, and HateXplain to ensure consistency and comparability with mainstream methods, such as GroupDRO, DFR, and Lu et al., which widely use these datasets to evaluate fairness through metrics like worst-group accuracy. These datasets have become standard benchmarks for assessing fairness in classification tasks and are particularly suited for calculating metrics that our method directly targets. Using the same datasets enables a fair and transparent comparison of FairLoRA's performance with these established approaches.

Alternative Datasets:

We acknowledge that datasets like Winogender, HolisticBias, and CrowSPairs are also fairness-focused and designed for classification tasks. However, our primary objective was to maintain comparability with prior work and leverage widely-accepted benchmarks in fairness research that specifically focus on biases caused by imbalanced sensitive attributes. Exploring these additional datasets in future work could provide complementary insights but was beyond the scope of the current study.

We hope this response addresses your concerns and clarifies the rationale behind our dataset choices.

---

Response to Q4:

The proportion of “male” in the CelebA dataset is 41.94%, whereas the proportion of “blond hair” is only 14.91%. The goal of our study is to mitigate bias introduced by the imbalance in sensitive attributes during training. By selecting “blond hair” as the sensitive attribute and “male” as the label, we ensure that the observed biases are due to the imbalance in the sensitive attribute, which aligns with the research objective of our work. Conversely, if we were to reverse this setup (i.e., using “male” as the sensitive attribute), the observed bias would primarily stem from the imbalance in the label rather than the sensitive attribute, which would not align with the issue we aim to address.

We hope this explanation clarifies our rationale, and we appreciate your attention to this detail.

Thank you for the detailed responses to my questions/comments!

Responses to Questions

Q1/Q2: That clarity makes sense, thanks!

Q3: The desire to maintain comparison to other works is also understandable. While it doesn't affect my scoring at all, I do suggest the minor rewording, as MNLI is not explicitly a fairness dataset.

Response to W1

Thanks for the additional perspective! However, I'm still unclear if and whether FairLoRA works on an unbalanced dataset where the ratio of majority to minority samples is not 1:1. This is far more likely for real world examples. Does Section 3.4 (L243-L245) hold when the finetuning dataset is not explicitly balanced?

Response to W2

It's helpful to see these additional experiments and that the results hold across other attributes. For clarity, I understand "Heavy Makeup" and "Wearing Lipstick" to be the sensitive attributes but what are the labels here?

Response to Comments

This also makes sense to me. Thanks for considering the terminology shift. With respect to using "Blond Hair" as the sensitive attribute, it sounds like this is a small terminology difference as well. I would generally argue that the sensitive attributes refer to protected groups (e.g. gender minorities). However, this is a small and nuanced point in the grand scheme and doesn't limit the contributions of your work.

Based on these responses, I've updated my score from 5 to 6.