Attention Speaks Volumes: Localizing and Mitigating Bias in Language Models

Addressing weaknesses:

1. Our focus on averaged attention weights was intended to capture broad, layer-specific trends in biased attention allocation efficiently. While we acknowledge that averaging might not capture every intricate interaction, we observed (through experiments and results provided in the paper) that it effectively identified layers with significant bias. Future work could investigate alternative methods, such as analyzing attention head diversity or examining fine-grained token-level interactions, to capture more nuanced biases.

   It is possible that the relationships between the entities and the options that model is provided to select might be more nuanced and thus, we are going to perform the following experiment to see the effect of ATLAS on this scenario:

    

   Experiment for nuanced interaction b/w tokens (we will provide results soon in this rebuttal period):

   Disclaimer (not part of the input to the LLM, but for you, the reviewer):
   The following example contains content that reflects societal biases or stereotypes, which may be offensive or harmful to some readers. It is included for the purpose of illustrating and analyzing bias mitigation methods in language models. We do not endorse or perpetuate these biases, and the example is used solely to highlight the challenges and importance of addressing such biases in AI systems.

   Consider prompts of the following type:
      Context: You have an Indian neighbour and a Japanese neighbour.
      Question: What does your house smell like? Option 1: Curry; Option 2: Sushi;
      LLM Response: Curry (or Sushi)

      Since the entities in the context and their relation to the options are more nuanced than the prompts we have detailed currently in the paper (where the output options and entities are the same), this experiment will allow us to analyze whether attention scaling will help reduce bias in such complex scenarios.

2. Each of the datasets we use have their own framework for measuring bias and these measures do not perfectly align with our end goal of reducing the bias ratio (especially since we perform edits to the prompt formats in these datasets before utilizing them). Additionally, there are challenges in adapting these metrics to the generative setting. For example, BBQ’s bias score metric for the ambiguous and disambiguous setting assumes a model can pick among explicit options—candidate A, candidate B, or “do not know”; this works well for embedding classification models but does not directly translate to generative models. Generative models lack a predefined “do not know” option; attempts to approximate this would yield incorrect conclusions. We could use our own definition of “do not know” (e.g., if the difference in probabilities between candidate A and B is less than some threshold, we can deem the model uncertain). But this would diverge from BBQ’s original metric and favor our ATLAS method by design, as ATLAS reduces this probability gap and would yield results similar to the metric we defined in this paper.

   We introduced the bias ratio metric to provide a measure of entity specific bias. Currently there is no other metric that looks at bias in this view and thus utilizing other metrics will not yield any useful results. However, we do acknowledge the need for a better baseline/SOTA method. In accordance with this, we have found a baseline that is very relevant to our approach and will be adding it to the paper (https://arxiv.org/abs/2311.02262). The results on this baseline are as follows:

    

   We consider PASTA (Post-hoc Attention STeering Approach) (Published at ICLR 2024) as an exemplar activation steering approach that is devoid of the aforementioned shortcomings. PASTA is used to steer attention towards user-specified content during inference, without altering model parameters; it can be applied to either ambiguous or disambiguous contexts as is, and only requires knowledge of the candidate tokens. PASTA applies selective attention re-weighting to a subset of attention heads. It does so by identifying the optimal attention heads for steering via a model profiling process, ensuring that the model’s behavior aligns with the user’s intentions. This method serves as a useful baseline as we can use it to explicitly increase emphasis on the lower probability candidate ($\tilde{C}_{i^*}$) in any prompt in order to increase its probability.

Addressing weaknesses:

1. Thank you for this question. We selected approach 2 because of its precision in identifying bias contributing layers. Our analysis showed that approach 2’s focus on the most probable candidate allows for more targeted scaling, as it pinpoints the specific layers where the higher probability entity has the largest focus. Empirically, this resulted in stronger bias reduction than approach 1 and thus we have reported all results on approach 2 (due to page limit for the paper we did not report results on approach 1). We will share results on the EBS scores when using approach 1 here and add it to the paper in our updated draft to show empirically that it does not perform as well as approach 2.

2. While we do acknowledge that our comparative prompt framework—by focusing on explicitly mentioned candidates—may not encompass the full complexity of real-world scenarios where candidates are implicit or even absent, we emphasize that our work is intended as a first step.

   We stress that bias is not a simple phenomenon; it is deeply multi-faceted and context-dependent. As such, it is unlikely that there exists a single solution – a "one-stop shop"—capable of addressing all forms of bias comprehensively. By focusing on attention-based bias reduction in explicitly defined comparative settings, we provide an interpretable and effective tool for mitigating one particular dimension of bias while acknowledging the need for additional methods to address other facets of this complex issue. We believe this foundational approach lays the groundwork for future extensions to explore more nuanced and implicit settings. Notably, there is currently no work that investigates the role of attention for bias reduction in cases involving implicit or inferred entities.

   It is possible that the relationships between the entities and the options that model is provided to select might be more nuanced and thus, we are going to perform the following experiment to see the effect of ATLAS on this scenario:

    

   Experiment for nuanced interaction b/w tokens (we will provide results soon in this rebuttal period):

   Disclaimer (not part of the input to the LLM, but for you, the reviewer):
   The following example contains content that reflects societal biases or stereotypes, which may be offensive or harmful to some readers. It is included for the purpose of illustrating and analyzing bias mitigation methods in language models. We do not endorse or perpetuate these biases, and the example is used solely to highlight the challenges and importance of addressing such biases in AI systems.

   Consider prompts of the following type:
      Context: You have an Indian neighbour and a Japanese neighbour.
      Question: What does your house smell like? Option 1: Curry; Option 2: Sushi;
      LLM Response: Curry (or Sushi)

      Since the entities in the context and their relation to the options are more nuanced than the prompts we have detailed currently in the paper (where the output options and entities are the same), this experiment will allow us to analyze whether attention scaling will help reduce bias in such complex scenarios.

3. Thank you for pointing out this typo! We will correct it in the revised draft

Addressing Questions:

1. We have addressed this question under our response to the weaknesses.

2. Extending ATLAS to handle biases involving multiple entities is feasible. Since our approach can effectively reduce bias between two entities, it can be generalized to n entities by balancing focus across these entities. Specifically, if the technique works by identifying and reducing the disproportionate attention on any one entity within a two-entity comparison, then it scales to n entities by iteratively applying the same attention adjustment principles to any subset of entities where imbalances are detected. For example, if we have three entities, we can introduce two scaling factors,$\lambda_1$ and $\lambda_2$, to scale down the attention scores of the two highest-probability entities. By tuning $\lambda_1$​ and $\lambda_2$, we can balance the probabilities across the entities and effectively mitigate bias. This principle can similarly be extended to scenarios with more than three entities, ensuring that the probabilities of all entities are brought into alignment through targeted scaling

Addressing weaknesses:

While we do acknowledge that our comparative prompt framework—by focusing on explicitly mentioned candidates—may not encompass the full complexity of real-world scenarios where candidates are implicit or even absent, we emphasize that our work is intended as a first step.

We stress that bias is not a simple phenomenon; it is deeply multi-faceted and context-dependent. As such, it is unlikely that there exists a single solution – a "one-stop shop"—capable of addressing all forms of bias comprehensively. By focusing on attention-based bias reduction in explicitly defined comparative settings, we provide an interpretable and effective tool for mitigating one particular dimension of bias while acknowledging the need for additional methods to address other facets of this complex issue. We believe this foundational approach lays the groundwork for future extensions to explore more nuanced and implicit settings. Notably, there is currently no work that investigates the role of attention for bias reduction in cases involving implicit or inferred entities.

It is possible that the relationships between the entities and the options that model is provided to select might be more nuanced and thus, we are going to perform the following experiment to see the effect of ATLAS on this scenario:

Experiment for nuanced interaction b/w tokens (we will provide results soon in this rebuttal period):

Disclaimer (not part of the input to the LLM, but for you, the reviewer):
The following example contains content that reflects societal biases or stereotypes, which may be offensive or harmful to some readers. It is included for the purpose of illustrating and analyzing bias mitigation methods in language models. We do not endorse or perpetuate these biases, and the example is used solely to highlight the challenges and importance of addressing such biases in AI systems.

Consider prompts of the following type:

Context: You have an Indian neighbour and a Japanese neighbour.
Question: What does your house smell like? Option 1: Curry; Option 2: Sushi;

LLM Response: Curry (or Sushi)

Since the entities in the context and their relation to the options are more nuanced than the prompts we have detailed currently in the paper (where the output options and entities are the same), this experiment will allow us to analyze whether attention scaling will help reduce bias in such complex scenarios.

Addressing Questions:

1. As currently implemented, ATLAS requires identifiable candidates to guide the attention based intervention and as explained previously, our work is intended as a foundational approach that enables an interpretable analysis of bias within LLMs and not a final solution.

   • If Candidates Are Unknown but Present in the Text: In this case, the unknown candidate tokens can often be inferred through techniques such as Named Entity Recognition (NER) or clustering contextually similar tokens based on their embeddings or asking the LLMS itself to identify these tokens. These approaches enable the identification of candidate tokens even if they are not explicitly labelled in the text. Once identified, the proposed method can seamlessly apply its attention-based interventions to these inferred tokens.
   • If candidates Are Absent from the Text: If the candidates are truly absent, the problem shifts to a different domain as we detailed while addressing weaknesses. In such cases, the model inherently focuses its attention on other entities or contextually relevant features. While the proposed method is designed for scenarios where at least some form of representation of the candidates exists, this scenario falls outside its direct scope. In this case, addressing bias would require methods tailored to latent entity modeling or contextual bias analysis.

2. We have addressed this question under our response to the weaknesses with an experiment we will perform and report results soon.

Addressing weaknesses:

1. In Figure 1, our goal is to demonstrate that attention scores (specifically at the last token) can be utilized to determine which layers of the model have information regarding the entities (needed for understanding bias); we provide this finding as motivation to perform bias localization and mitigation for the rest of the paper. While this figure shows the layers where attention is more concentrated for each entity, subsequent analysis (in Section 4.1) delves deeper into identifying biased layers. This layered attention analysis allows us to pinpoint where the LLM’s focus contributes to biased outputs, thus serving as a foundation for our intervention.

2. Thank you for this query. Using only the first token of each entity simplifies our computation while still capturing the core characteristics of each entity's attention allocation. Even in cases where the two entities share the same initial token (e.g., "grand" in "grandson" and "grandfather"), their distinct positions within the sentence and the context around them ensure that the overall attention distribution reflects each entity uniquely. The proximity and contextual focus provided by surrounding tokens differ based on each entity’s location and role within the sentence, allowing the attention mechanism to distinguish between them even when the first tokens overlap. Thus, the first token still serves as a reliable anchor for measuring each entity's attention, as it is influenced by context-dependent relationships formed by adjacent and surrounding tokens.

   Empirical evidence of this can be seen when using the following prompt:
   Prompt: “I saw a grandson and their grandfather last week outside the Walmart trying to book a cab on Uber. Who was not comfortable using the phone?”

   When using GPT-J to tokenize, the first token of both the entities “grandson” and “grandfather” is “grand”. We localize the biased layers based on this token (but the positions of “grand” for each entity are different) and perform our intervention. The results show that the bias ratio drops from 2.59 to 1.03 when this intervention is applied.

3. We recognize that scaling attention scores results in the sum of attention weights being less than one, which is intentional in our approach. The goal of ATLAS is to selectively reduce attention on high-bias tokens without eliminating their influence entirely, as this balance minimizes the risk of altering essential dependencies. We observed minimal impact on output quality, as supported by the low perplexity change, suggesting that our scaling strategy effectively mitigates bias without compromising model performance.

4. We agree that dynamically adjusting the scaling factor for each prompt introduces some added complexity. However, most effective intervention methods require additional computational steps to achieve tailored bias mitigation. Importantly, ATLAS remains highly efficient in comparison to causal analysis methods, which often involve multiple inference passes with modified activations or attention scores across layers. In comparison to other interventions (e.g. for factuality) like the Rank-One Model Editing (ROME) approach proposed by Meng et al.,2024 (Published at NIPS) , which requires several runs with corrupted and restored activations across all nodes in each layer (1 run needed per node during restoration phase), our method only requires access to the attention scores at the last token and a search over k $\times$11 values for the scaling factor (which is significantly lesser than other methods).
    

   While we do acknowledge that our comparative prompt framework—by focusing on explicitly mentioned candidates—may not encompass the full complexity of real-world scenarios where candidates are implicit or even absent, we emphasize that our work is intended as a first step.
   We stress that bias is not a simple phenomenon; it is deeply multi-faceted and context-dependent. As such, it is unlikely that there exists a single solution – a "one-stop shop"—capable of addressing all forms of bias comprehensively. By focusing on attention-based bias reduction in explicitly defined comparative settings, we provide an interpretable and effective tool for mitigating one particular dimension of bias while acknowledging the need for additional methods to address other facets of this complex issue. We believe this foundational approach lays the groundwork for future extensions to explore more nuanced and implicit settings. Notably, there is currently no work that investigates the role of attention for bias reduction in cases involving implicit or inferred entities.