Debiasing Algorithm through Model Adaptation

Thank you for your detailed comments and for recognizing the importance of debiasing in LLM research. We organize the rebuttal in three parts to comprehensively answer the raised concerns related to: 1) LM performance, 2) clarifications about concepts and equations in section 4; 3) the novelty of our approach.

Firstly, we want to address the concern about LM performance that might be an effect of misunderstanding our ablation setting:

Figure 3 (b) seems to indicate that to reduce the bias, the cost is to get a much worse language model.

Figure 3 (b) illustrates ablation experiments with varying intervention layer numbers. The significant drop in perplexity occurs notably for 10 or 11 layers. It's crucial to note that, as stated in the beginning of section 4.4: "We apply DAMA to MLPs in approximately one-third of the model’s upper layers (in LLaMA 7B layers 21 - 29 out of 32),” equating to 9 layers. The corresponding perplexity is denoted with a star on the figure, which is close to the score of the original model.

Secondly, we address your questions regarding concepts in Section 4. Moreover, we refined the section for better clarity and uploaded it to open review.

"Certain concepts [..] are either inconsistent or not explicitly defined."

We clarified in section 4 that:

• “stereotypical keys” and “stereotyped keys” are the same entity. We use just the former name.
• “gender values” and. “gendered values” are the same entity. The latter name is now used exclusively.
• “Grammatical gender” is the quality manifested by the specific inflection of words. We now use a descriptive definition of grammatical gender to avoid confusion with gender scores.

Questions regarding Equation 2

$z$ is the latent vector, specifically the output of the lth feed-forward layer, as denoted in the subscript $FF_{out,l}=z$. We used it just as a loss argument to find the gendered value vector $v_o$ We added $\text{argmin}$ notation in Equation 2 to clarify that. The missing parenthesis was also fixed.

"The first two items in equations look similar to variational inference."

In equation 2, we consider the model's output probability $P(o'|X')$ conditioned on a known prompt ( $X'$). The formula resembles variational inference, yet there is a crucial difference: $X'$ is a known prompt, not a random variable. As a result, a prior distribution cannot be defined here except for a trivial one.

Meaning of metrics $U$,$V$, and $P$:

We just want to clarify that $U$ and $V$ are matrices of concatenated stereotypical key vectors and gendered value vectors as described in the article (those are not metrics). We added further explanation regarding equation 5 in the revised paper.

Lastly, we’d like to highlight the novelty of DAMA presented in the Discussion Section.

Novelty in the context of model editing methods

Previous model editing methods targeted highly specific information encoded in the model ( Mitchell et al., 2022; De Cao et al., 2021), while we introduced a method capable of editing general dataset artifacts, such as various manifestations of gender bias. In Table 2, we show that DAMA significantly outperforms MEMIT (Meng et al., 2022 ) in this aspect.

Advancements in projection-based debaising.

In contrast to earlier projection-based debiasing methods that focused on learning and projecting latent representations for specific gender-related tasks (e.g., gender classification in bios Ravfogel et al. 2020), our method demonstrates that projection can be achieved without relying on auxiliary tasks. Notably, the projections in our approach are directly applied to the model component, avoiding alterations to the architecture or parameter size. Our theoretical and empirical evidence further establishes that applying debiasing projections on MLPs effectively reduces bias while preserving other encoded information, showcasing the generalizability of the method across tasks.

We hope that the response addresses most, if not all, of your concerns. We are open to further discussion, suggestions, or questions you may have.

Thank you for your insightful review, particularly for acknowledging the significance of evaluating diverse gender bias metrics and downstream tasks. We also appreciate your positive evaluation of our method's theoretical background.

Seems like there is still quite some room for improvements

We agree that the method can be further improved for larger scales. In our opinion, the crucial strength of DAMA is its versatility in reducing bias for professions and tasks unseen in the adaptation. This contrasts with many past works that did not show consistent patterns across tasks (Delobelle et al. 2022, van der Wal et al. 2023). Admittedly, bias annotation is oftentimes not consistent. Thus, some manifestation of it may always be present in the model when not specifically targeted.

Is there any hypothesis on why it doesn't fare well with the reasoning type of tasks? ARC

Please note that the difference between model instances in Table 2 is only barely statistically significant. We compared the errors on the ARC-C task made by the original 65B model and the same model with DAMA and did not observe any clear patterns. On average, the original model selects the correct answer for 40% of DAMA’s errors, while DAMA picks the correct answer for 33% of the original model’s errors. Considering the probability of randomly choosing the correct answer is 33%, it suggests that the performance difference may be attributed to a random factor.

You are missing a right parenthesis on equation (2) for the KL divergence.

The equation was fixed in the updated version of the paper.

Thank you once more for your review. Should you have any additional comments or questions, we'll be happy to address them.

Thank you for your thoughtful comments and positive review of our work and for recognizing the importance of the proposed approach. We appreciate your acknowledgment of the need to evaluate the debiased model on a wide range of bias metrics and downstream tasks.

In this response, we want to highlight the novelty of DAMA and further explain our choices in the experimental setting.

Novelty of DAMA

Novelty in the context of model editing methods and comparison to MEMIT

Previous model editing methods targeted highly specific information encoded in the model ( Mitchell et al., 2022; De Cao et al., 2021), while we introduced a method capable of editing general dataset artifacts, such as various manifestations of gender bias. Specifically, we show that the intervention performed on a set of professions generalizes well to unseen professions and even other manifestations of Gender Bias. This is beyond the capabilities of MEMIT (Meng et al. 2022), which targets only specific pieces of information that are sparsely encoded. Hence, MEMIT intervention shows marginally better performance on downstream tasks while still keeping bias in the model.

Advancements in projection-based debaising.

In contrast to earlier projection-based debiasing methods that focused on learning and projecting latent representations for specific gender-related tasks (e.g., gender classification in bios Ravfogel et al. 2020), our method demonstrates that projection can be achieved without relying on auxiliary tasks. Notably, the projections in our approach are directly applied to the model component, avoiding alterations to the architecture or parameter size. Our theoretical and empirical evidence further establishes that applying debiasing projections on MLPs effectively reduces bias while preserving other encoded information, showcasing the generalizability of the method across tasks.

Further explanation for experimental setting choices:

It is unclear why a linear model is fitted across the two gender scores in order to investigate the extent of bias.

Thank you for this insightful observation. We have also checked the linear coefficients of the linear models fitted on one of the gender scores (factual and stereotypical). Notably, the patterns are closely aligned with those observed in the joint linear model, visible in the table below. We opted to present the coefficients of a joint model in the paper because it has one intercept coefficient, which has a clear interpretation as the language model’s preference for male pronoun when the subject is marginalized (as noted at the end of section 2.3).

Moreover, Pearson’s correlation between factual and stereotypical scores is moderately low: $\rho=0.258$ yet statistically significant.

only focusing on Llama, as it is unclear how much these findings generalize to other models.

We intend to explore the applicability of DAMA with other models and languages in future work. We chose LLaMA because of its strong performance and availability of weights across different scales.

Once again, thank you for your review. We will be happy to answer any further questions regarding our method.