Does Example Selection for In-Context Learning Amplify the Biases of Large Language Models?

We appreciate the reviewer’s recognition of the significance of our work. Thank you for your valuable and thoughtful comments. Please find our point-by-point responses below:

Weakness 1 (Validation of EEC-paraphrase)

A major methodological weakness lies in the dataset construction. Using GPT-3.5-Turbo to create EEC-paraphrase raises questions about potential inherited biases and quality control.

Thank you for your valuable feedback. First, for quality control, previous work has demonstrated the capability of LLMs (including GPT-3.5-Turbo) to produce diverse and valid paraphrases under guidance [1]. We further validate the quality of EEC-paraphrase with the help of the Python library Textstat and provide results in the table below.

On the other hand, we have introduced a new task - toxicity detection, and believe that multiple tasks and multiple LLMs can minimize the impact of these potential biases on the results. We have also sampled and manually checked the generated samples.

The table below shows the mean and standard deviation of the performance of datasets on various metrics. We choose the number of words and Distinct-n to measure the diversity of sentences, and metrics in Textstat are used to measure the complexity of sentences, which help determine the readability, complexity, and grade level.

[1] ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustness (Cegin et al., EMNLP 2023)

Questions

Regarding the question 1

Please refer to the response to Weakness 1.

Question 2 (Additional task)

Have you explored whether these patterns hold true for other tasks like toxicity detection or text classification? Even preliminary results would help assess generalizability.

We really appreciate your suggestions. Following your valuable guidance, we have supplemented the test of LLMs for toxicity detection using Jigsaw as the dataset and present the results for the maximum values of gender bias below.

In toxicity detection, LLMs are asked to judge whether the sentences given are toxic or non-toxic. In the table below, we provide the gender bias performance of Llama-2-7B and Llama-3.2-3B in toxicity detection. To stay clear, the results below are the maximum values of gender bias, and the remaining details (such as the mean values) are available in the revised appendix.

Consistent with the sentiment analysis, we can find that:

• Compared with the zero-shot, example selection methods for ICL amplify the maximum value of gender bias;

• ReBE remains compatible with example selection methods and exhibits effective debiasing in toxicity detection.

Weakness 2 (Baseline comparison)

The absence of comparisons with existing debiasing methods and fine-tuning approaches makes it difficult to fully assess ReBE's advantages.

Thank you for your valuable feedback. First, we would like to clarify that fine-tuning methods are not suitable for comparison with ReBE. Our work focuses on bias risks of example selections for ICL and debiasing methods that can serve ICL. However, fine-tuning approaches need to update the LLM parameters, which destroy the advantages of ICL.

Second, there are a few debiasing methods specifically for ICL (Note that, they are NOT specifically for our research problem). Although Hu et al. [1] proposed a fairness via clustering genetic (FCG) algorithm, FCG needs explicit feature vectors to complete the clustering. Due to this limitation, FCG cannot apply to sentiment analysis and toxicity detection, so we cannot set it as a baseline for ReBE. Even so, we value the reviewer's opinion and compare ReBE with two context augmentation methods: Counterfactual and gender-balanced.

• Counterfactual

  For a dataset built based on templates like EEC, it is convenient to construct the corresponding counterfactual instance according to the templates. For example, according to the template <person subject> feels <emotion word>, the counterfactual instance of sentence Alonzo feels angry. can be Nichelle feels angry. or Amanda feels angry..

• Gender-balanced

  The gender-balanced context approach requires an equal or close number of examples for each gender type.

To stay clear, we present the results of random-based and DPP-based example selections in the tables below. More details are available in the revised appendix.

The following table shows the gender bias of OPT-6.7B on Sentiment Analysis with EEC-paraphrase as the dataset. While the EEC-paraphrase dataset does not have its own templates, it is built upon the EEC samples, allowing us to generate counterfactual samples using the templates from the EEC.

Since the counterfactual context method does not apply to datasets without templates, we remove it from the baselines tested on the Jigsaw dataset. The following table shows the gender bias of Llama-2-7B on Toxicity Detection with Jigsaw as the dataset.

Takeaways:

• There is currently NO suitable debiasing baseline specifically for ICL to compare with ReBE;
• Compared with the counterfactual and gender-balanced context method, ReBE is compatible with existing example selection methods and can achieve lower bias and higher accuracy.

[1] Strategic Demonstration Selection for Improved Fairness in LLM In-Context Learning (Hu et al, EMNLP 2024)

Dear Reviewer Av69,

Thank you again for the valuable comments that helped improve our paper. Following your suggestions, we have carefully revised the manuscript to address your concerns.

We kindly remind you to review our reply along with the revised submission and consider updating your evaluation accordingly.

Best,

Authors

Dear Reviewer Av69,

We appreciate all of the valuable time and effort you have spent reviewing our paper. As today is the last day of the discussion period, we gently request that you review our reply and consider updating your evaluation accordingly. We believe that we have addressed all questions and concerns raised, but please feel free to ask any clarifying questions you might have before the end of the discussion period.

Best,

Authors

[W4.2] The authors haven't done any comparison to simpler bias mitigation approaches when many exist.

Thank you for your valuable feedback. We would like to clarify that we have presented a discussion of other debiasing methods in Subsection 3.5 (Appendix G.1 of the current revision).

First, there are a few debiasing methods specifically for ICL (Note that, they are NOT specifically for our research problem). Although Hu et al. [1] proposed a fairness via clustering genetic (FCG) algorithm, FCG needs explicit feature vectors to complete the clustering. Due to this limitation, FCG cannot apply to sentiment analysis and toxicity detection, so we cannot set it as a baseline for ReBE. Second, we value the reviewer's opinion and compare ReBE with two context augmentation methods: Counterfactual and gender-balanced.

• Counterfactual

  For a dataset built based on templates like EEC, it is convenient to construct the corresponding counterfactual instance according to the templates. For example, according to the template <person subject> feels <emotion word>, the counterfactual instance of sentence Alonzo feels angry. can be Nichelle feels angry. or Amanda feels angry..

• Gender-balanced

  The gender-balanced context approach requires an equal or close number of examples for each gender type.

To stay clear, only the results of random-based example selection are shown below. More details are available in the revised appendix.

The following table shows the gender bias of OPT-6.7B on Sentiment Analysis with EEC-paraphrase as the dataset. While the EEC-paraphrase dataset does not have its own templates, it is built upon the EEC samples, allowing us to generate counterfactual samples using the templates from the EEC.

Since the counterfactual context method does not apply to datasets without templates, we remove it from the baselines tested on the Jigsaw dataset. The following table shows the gender bias of Llama-2-7B on Toxicity Detection with Jigsaw as the dataset.

Takeaways:

• There is currently NO suitable debiasing baseline specifically for ICL to compare with ReBE;
• Compared with the counterfactual and gender-balanced context method, ReBE is compatible with existing example selection methods and can achieve lower bias and higher accuracy.

[1] Strategic Demonstration Selection for Improved Fairness in LLM In-Context Learning (Hu et al, EMNLP 2024)

Questions:

[Q1] I'd like you to do some dataset validation, such as through human evaluation of the paraphrased sentences to ensure quality and meaning preservation, comparison with real-world text samples to validate ecological validity, and analysis of potential artifacts introduced by GPT-3.5 paraphrasing.

Please refer to the response to Weakness 1.

[Q2] Please include ablation studies isolating each component of ReBE. I'd prefer to see some comparison with simpler debiasing approaches as well.

Please refer to the response to Weakness 2.1 and Weakness 4.2.

[Q3] Could you come up with similar datasets for different types of tasks beyond sentiment analysis and conduct experiments there as well?

Please refer to the response to Weakness 4.1.

[Q4] There are many issues with the presentation.

Please refer to the response to Weakness 3.

Weakness 2 (Ablation study and significance test)

No ablation studies showing which components of ReBE are actually responsible for bias reduction.

Thank you for your valuable feedback. We would like to clarify that we included the ablation study in subsection 5.2 and provided the experiment results in Table 4.

As for the components responsible for bias reduction, since ReBE introduces contrastive loss based on prompt tuning, components that can be used for ablation experiments are the loss functions.

In subsection 5.2, we tested the cases of $L_{acc}$ and $L_{bias}$, which represent the cases of not using $L_{bias}$ and not using $L_{acc}$, respectively. Based on the results in Table 4, we have revised and concluded that $L_{bias}$ is actually responsible for bias reduction, and $L_{acc}$ guarantees accuracy (lines 444-446).

The authors haven't shared any statistical significance tests in the paper.

Thank you for your valuable feedback. We would like to clarify that statistical significance test is not suitable for verification of results in this paper.

The statistical significance test can help determine whether there is a difference between two data sets and whether the difference is significant. In other words, it estimates the probability that two data sets belong to the same overall distribution. However, because the inputs of LLM in ICL are constructed based on the same dataset and belong to the same distribution, the outputs of the same LLM under various example selection methods still belong to the same overall distribution. Therefore, the statistical significance test is not applicable to the comparison between different example selection methods.

Why do the inputs belong to the same distribution ?

In ICL, each input consists of a question and a context, which is a combination of $k$ samples. Although different example selection methods select different $k$ samples to construct the context, these samples belong to the same dataset. When the context distribution of random-based example selection include enough points, other example selection methods can be understood as sampling part of this distribution according to specific rules. So, theoretically, inputs of LLM under various example selection methods belong to the same overall distribution.

Why is the statistical significance test also not applicable to the comparison of ReBE ?

Since ReBE is implemented based on prompt tuning without updating LLM parameters, the above discussion on output distribution still applies to the combination of ReBE and other example selection methods. More specifically, ReBE adds a small amount of virtual tokens to the original input. Compared with the input length (1024) of LLMs, the number of virtual tokens is tiny (10-30), so the output distribution of ReBE could be close to the example selection methods, the statistical significance test is not applicable to the comparison between ReBE and other example selection methods.

What did we do to ensure the reliability of the results ?

To reduce the randomness of results, we followed the advice and tested nine LLMs on two tasks (sentiment analysis and toxicity detection) under multiple random seeds.

We sincerely appreciate your comments. Please find our point-by-point responses below:

Weakness 1 (Validation of EEC-paraphrase)

There is no validation for the claim that EEC-paraphrase dataset consists of sentences that are more complex and natural.

Thank you for your valuable feedback. To quantitatively demonstrate the complexity and naturalness of the EEC-paraphrase dataset, we evaluate using the Python library Textstat and present the results in the table below.

In selecting metrics, we choose the number of words and Distinct-n to measure the sentence diversity, which reflects the naturalness. To evaluate complexity, we use metrics from Textstat to determine the readability, complexity, and grade level of sentences.

The results show that the EEC-paraphrase dataset outperforms the original EEC dataset across all metrics.

Using an LLM to create sentences that are then used to study biases in other LLMs is not the right way to design this experiment.

Thanks for your valuable feedback. First, we argue that the statement "Using an LLM to create sentences that are then used to study biases in other LLMs" is not accurate because EEC-paraphrase is not created out of thin air by GPT-3.5-Turbo, but paraphrase the existing dataset EEC.

Second, previous work has demonstrated the capability of LLMs (including GPT-3.5-Turbo) to produce diverse and valid paraphrases under guidance[1]. Therefore, we think that leveraging LLMs, under human review, to assist in part of the dataset construction process—rather than relying on crowdsourcing—is a reasonable experimental design. We follow the practices adopted by previous work.

[1] ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustness (Cegin et al., EMNLP 2023)

Weakness 3

The paper is not well written and the exposition needs a lot of improvement.

Thank you for your valuable feedback. We have tried our best to polish the language in the revised manuscript. We would be grateful if you could provide more specific examples.

Weakness 4.1 (Additional task)

All experiments are on a single task type (sentiment analysis).

Thank you for your valuable feedback. We have supplemented the test of LLMs for toxicity detection using Jigsaw as the dataset and present the results for the maximum values of gender bias below.

In toxicity detection, LLMs are asked to judge whether the sentences given are toxic or non-toxic. In the table below, we provide the gender bias performance of Llama-2-7B and Llama-3.2-3B in toxicity detection. To stay clear, the results below are the maximum values of gender bias, and the remaining details (such as the mean values) are available in the revised appendix.

Consistent with the sentiment analysis, we can find that:

• Compared with the zero-shot, example selection methods for ICL amplify the maximum value of gender bias;
• ReBE remains compatible with example selection methods and exhibits effective debiasing in toxicity detection.

Dear Reviewer wJ1K,

Thank you again for the valuable comments that helped improve our paper. Following your suggestions, we have carefully revised the manuscript to address your concerns.

We kindly remind you to review our reply along with the revised submission and consider updating your evaluation accordingly.

Best,

Authors

Thank you for your valuable feedback. First, we think we have addressed and responded to most of the concerns in our rebuttal above. Summarize them as follows.

EEC-paraphrase dataset validation

We have addressed it. Please refer to $\rightarrow$ Author Response - Part 1: Weakness 1 (Validation of EEC-paraphrase)

Ablation studies

We have addressed it. Please refer to $\rightarrow$ Author Response - Part 3: Weakness 2 (Ablation study and significance test)

Second, we would like to provide further explanations for some misunderstandings.

The substantial difference in MaxFG values between sentiment analysis (0.2-0.4) and toxicity detection (up to 0.857) suggests task-dependent performance.

It is unreasonable to compare the debiasing results without considering the original data and task differences.

This reasoning is flawed because we're not interested in whether the outputs come from the same distribution---we want to know if the differences in performance metrics (accuracy, bias measures) between methods are statistically reliable.

Because the performance metrics are calculated completely based on the outputs, we think the reasoning is valid.

Some suggestions for improving presentations

Thank you for your valuable suggestions; since the reviewer's latest reply is after the revision submission deadline, we are unable to upload the improved manuscript according to the suggestions. However, we will include the corresponding revisions in subsequent updates.

Thank you again for your valuable feedback!

Dear Reviewer wJ1K,
Thank you for your reply. We do believe our rebuttal has addressed the concerns you posted in your comments. We also clarified some misunderstandings that you may have. For a better discussion, could you please clearly specify the specific technical points of your concerns that you have? Greatly appreciate that!

We thank the reviewer for your recognition of our work. We also appreciate the detailed comments posed by the reviewer. Please find below the point-to-point responses to your comments.

Weakness 1 (The claim may be too bold)

The claim that “all LLMs exhibit an increase in the maximum gender or race bias value with random-based example selection for ICL” (lines 217-218) may be too bold of a claim based on their mixed results of bias amplification.

Thank you for your valuable feedback. To make the wording more rigorous, we have corrected the claim as "the LLMs tested exhibit varying degrees of increase in the maximum gender or race bias value with random-based example selection for ICL" (lines 223-224).

Besides, we would like to give a brief explanation as to whether the claim is too bold. Although some LLMs do not exhibit simultaneous amplification of the maximum values of gender and race bias in sentiment analysis, we do not argue that the claim does not apply to these models. This is because the experiment results are task-dependent, and our additional experiment results for toxicity detection support this.

For instance, in sentiment analysis, Llama-2-7B does not show an amplification of the maximum value of gender bias, but it does exhibit significant gender bias in toxicity detection (Detailed data is available in the revision appendix). Therefore, as long as the LLM shows amplification of the maximum value of a single type of bias, we think the claim applies to it.

Weakness 2 (Limited bias mitigation over DPP)

While ReBE reduces bias in many cases, its improvements over DPP are limited (Figure 7).

Thank you for your valuable feedback. Firstly, we acknowledge that reducing bias becomes increasingly challenging when the initial level of bias is already minimal. Compared with other example selection methods, the bias values of DPP-based example selection are smaller (The max value of AvgGF is just 7% in Figure 6, which is the Figure 7 of the previous submission). Additionally, it is generally understood that the goal of a debiasing method is to reduce bias to an acceptable range, as completely eliminating it is often difficult to achieve.

Weakness 3 (Impact of $k$)

There is no analysis on the impact of increasing the number of ICL examples.

Thank you for your suggestions. We have added the corresponding experiment results and analysis in Appendix E. To investigate the impact of increasing the number of IC examples, we have assessed the gender bias performance of Llama-2-7B in toxicity detection under various number of ICL examples ($k\in[2,6,10,14,18,22,26]$).

1. Gender bias performance of Llama-2-7B on $AvgGF$

2. Gender bias performances of Llama-2-7B on $MaxTG$ and $MaxFG$ are available in the revision appendix.

Findings: As the number of ICL examples $k$ increases, the bias decreases overall, but the change in accuracy must also be considered. Please see the revision for detailed additional data.

Question 1 (Confusing Figure 6)

Figure 6 is a little confusing and is not fully explained either in the main body of the paper or the Figure caption.

We apologize for the confusion. We have added more detailed explanation and hope this can resolve your confusion (lines 342-345).

Question 2 (Impact of $k$)

How does increasing the number of ICL examples impact bias in other example selection methods?

Please refer to the findings in the response to Weakness 3.

Question 3 (Additional train time)

The additional train time required to add ReBE to existing methods.

That's a good question, we give an example in the following table.