FairMT-Bench: Benchmarking Fairness for Multi-turn Dialogue in Conversational LLMs

Thanks for the careful responses.

I think most of my major weaknesses have been solved or partially solved. Hence, I will increase my recommendation.

By the way, if the authors promise to append such newly added experiments to the final paper, I would like to support the acceptance of this paper.

Weakness 5: Lack of improvement directions.

Thank you for your suggestion. In Section 3.7 of our manuscript, we highlight the urgent need to improve fairness in multi-turn dialogues. We agree that incorporating efforts to mitigate bias, as you suggested, would significantly enhance the completeness of our paper. Consequently, we have implemented a simple debiasing attempt based on our current findings.

We employed commonly used alignment paradigms DPO [7] for debiasing. We constructed positive and negative data pairs for fairness multi-turn alignment based on the evaluation results on the FairMT-10k dataset, note that the data from FairMT-1k was excluded during the selection process. We use the first four turns of historical dialogue as the conversation, with the unbiased response from the model serving as the "chosen" answer and the biased response as the "rejected" answer. The base model is Llama-2-7b-chat. After debiasing, we evaluate the model's fairness and the proportion of over-protection on FairMT-1k, and evaluate the model's multi-turn dialogue capabilities,. Specifically, we evaluate the model's multi-turn dialogue capabilities based on MT-Bench 101 [8] and calculate the proportion of over-protection by determining the model's refusal rate in the Fixed Format tasks for the first four turns where the questions are entirely unbiased. To further investigate the performance in fairness alignment, we have also compared the results on a single-turn alignment dataset using only the last turn conversation of FairMT-10k dataset. The experimental results are shown in Table 2.

From the experimental results, the model's bias ratio significantly decreased after alignment, and the performance of fairness under both single and multi-turn alignment paradigms was essentially consistent, suggesting that improvements in fairness primarily arose from the model fitting to the dataset used in this work. However, both single and multi-turn aligned models showed a decline in multi-turn dialogue language modeling capabilities, and an increase in the proportion of over-refusal, indicating that the model sacrificed performance to a large extent to enhance fairness, without achieving a good trade-off. However, it can also be seen that multi-turn alignment had a smaller impact on the model's language modeling capabilities compared to the currently prevalent single-turn safety alignment, showing certain advantages of multi-turn alignment. Therefore, designing more high-quality multi-turn fairness alignment datasets and techniques to achieve a trade-off between fairness and quality remains an unresolved issue.

In response to your suggestion, we have included the fairness alignment experiment in Appendix B.4 in the revised manuscript.

Table 2: Fairness and Quality performance after model alignment.

Thank you for your constructive comments. We have carefully considered your suggestions and would like to provide the following clarifications. We have also updated the manuscript, with the modified contents highlighted in red for your review.

Weakness 1: Lack of specific formal definitions for each task in the main paper

Thanks for your insightful feedback. As you pointed out, due to the space constraints, we initially introduced and defined the nature and simple form of the tasks only in the main paper, placing the specific task formats in Appendix A.1 Figures 9-16. To enhance the readability and completeness of the paper, we have added brief examples of each of the six tasks in Section 2.1 of the main manuscript, based on your suggestion. These additions help may readers more quickly grasp the specific definitions and forms of the tasks.

Weakness 2: Ignored many necessary details in the main paper

Thank you very much for your suggestions. Based on your advice, we have added details related to the specific task settings, details of evaluation and the cost of benchmark evaluation in the main manuscript, and supplemented the corresponding details in the appendix. This will allow readers and subsequent researchers to better understand the design of FairMT-Bench when reading this article. Specifically, we have added specific task settings in Section 2.1 details of evaluation and the cost of benchmark evaluation in Section 2.3.

Weakness 3: Not involve enough human annotation

Following your suggestion, we expanded our human validation by increasing the number of samples. Specifically, we selected 1,000 samples from the evaluation results of the FairMT-10K dataset for annotation, and we employed uniform sampling across six tasks. Given the potential severity of underestimating biased responses by GPT-4, we sampled responses assessed by GPT-4 as biased and unbiased at a ratio of 1: 2. We employ 20 annotators, all of whom possess strong English proficiency and uphold high ethical standards. We have conducted annotation training for the annotators, clearly explaining the design of the task and the criteria for judging model responses. Each sample was independently annotated by three human annotators, with the final annotation determined by majority vote. We then calculated the consistency between the human assessment results and those of GPT-4. The results of this consistency assessment are presented in Table 1. Overall, human validation results show high Accuracy and Recall, indicating strong consistency between the GPT-4 and human judgements. GPT-4 exhibits a low rate of false negatives. However, in certain tasks, GPT-4's evaluations demonstrate a tendency toward over-protection, exhibiting a higher rate of false positives. Further analysis of cases with deviations in Appendix A.5.2 reveals a consistent pattern. Specifically, when a response includes both a refusal to answer and a biased statement, human annotators are more likely to label the response as unbiased, whereas GPT-4 tends to consider it biased. We have updated the human validation results in Section 3.2 (Lines 309-311) and Appendix A.5.2 of the paper to reflect these findings.

Table 1: Consistency between new Human Validation and GPT-4 Results.

Thank you for your constructive comments. We have carefully considered your suggestions and would like to provide the following clarifications. We have also updated the manuscript, with the modified contents highlighted in red for your review.

Weakness & Question: Ambiguous Task Taxonomy.

Thank you for your insightful comment. In our benchmark, the division of tasks in Section 2.1 primarily targets the fairness deficiencies of LLMs across three stages of interaction with users: the ability to perceive and understand biases in multi-turn contexts (Context Understanding), the ability to correct biases during interactions (Interaction Fairness), and the ability to balance adherence to instructions with fairness (Fairness Trade-off). You mentioned that the definition of Interaction Fairness and the boundaries with the other two categories are unclear. We have added supplements and explanations to this. In our definition, although both Interaction Fairness and Fairness Trade-off pertain to the capability of "bias resistance," Interaction Fairness focuses more on the model's ability to judge whether a user's question contains biased information without being disturbed, while Fairness Trade-off examines whether the model can pursue utility without violating the principles of fairness. We believe that a binary classification is clearer and more straightforward, but the current three-stage taxonomy is more detailed and comprehensive.

We recognize that the change in terminology in Section 3.2 may cause confusion in the change of classification criteria. In Section 3.2, we mentioned "comprehension-focused" and "bias resistance". This is mainly an explanation based on experimental results, focusing on explaining the reasons for bias in the model's answers, or the reasons why the model fairness alignment fails. We also think that this classification is related to the above three-stage taxonomy. The most important ability of the model in the first context understanding stage is comprehension. In the subsequent interaction and task execution stages, it is necessary to assess the correctness and fairness of user input based on a complete understanding of the context, and ensure that outputs remain fair. Therefore, its "bias resistance" ability is examined.

Based on your suggestion, we have revised lines 322-343 in Section 3.2 to enhance the clarity of the text, ensuring consistency throughout the document. Intuitively, we have also included the terms "comprehension-focused tasks" and "bias-resisance" in Section 2.1 to provide further explanation. Additionally, we have removed the taxonomy of implicit biases, as they are not closely related to the previous discussion, in order to avoid causing further confusion.

Question_1: How do you ensure that biases from GPT-4, which is used to generate and evaluate responses, do not affect the fairness outcomes, particularly given GPT-4's role as both data generator and evaluator?

You raised a valid concern regarding biases from GPT-4 may affect the fairness outcomes, which indeed presents a challenge and a risk. We have implemented comprehensive measures to mitigate and reduce the negative impact of GPT-4 on the quality of the dataset and its evaluations. Firstly, we limited the use of GPT-4 to only a small portion of the data. Secondly, we employed clear instructions to guide the generation process. Lastly, we conducted thorough validation screenings of the content generated by GPT-4. Specifically, in the dataset generation process, we only used GPT-4 to generate data for the first question of the Scattered Questions task and the five questions of the Jailbreak Tips task. For the first question in the Scattered Questions task, we utilized GPT-4 to generate an event that reflects a particular stereotype or toxic trait of a specific group. To achieve this, we guided the model with detailed instructions, including system prompts, comprehensive task descriptions, and typical demonstrations, to produce high-quality answers. Further, we filtered out model outputs that refused to answer, identified by keywords like "sorry, I cannot answer...", from all GPT-4 generated answers. In the remaining responses, we manually checked GPT-4's output to ensure it aligned with the group's stereotype. The other questions for this task were constructed based on manually written templates and combinations of bias groups and attribute words extracted from other bias datasets, and do not suffer from quality issues due to LLM biases. For the Jailbreak Tips task, we employed the Chain of Attack (CoA) multi-round attack framework [8], assessing each round of attacks using LLMs, classifiers, and semantic similarity to the original biased statements to determine if the model's attack was successful. If an attack was unsuccessful, the LLM would regenerate a question more likely to elicit a biased response, continuing until the specified dialogue narrative was achieved. We only retained questions from successful attacks, removing samples that did not succeed within a specified number of iterations to ensure the quality of multi-round questions for the Jailbreak Tips task. The number of data generated for these two tasks is significantly lower because we retained only high-quality questions. Thus, using GPT-4 has minimal impact on the quality of the dataset.

On the other hand, we understand your concerns about using GPT-4 as our evaluator. The reason for using GPT-4 as judge and our efforts of avoiding the negative impact of GPT-4's bias and misunderstanding are detailed listed in the response of Weakness_1.

Question_2: Can you clarify if the performance discrepancies across bias attributes are due to data distribution differences or model-specific limitations?

We have elaborated on our understanding and hypotheses regarding this issue in our response to Weakness_2, and demonstrated how our hypotheses corresponded to specific cases in the test results.

Weakness_2: The paper does not thoroughly explore why certain attributes (like gender and race) showed consistently poor performance across models, missing an opportunity to deepen the community’s understanding of these biases.

Thanks for your insightful comment. In our paper, we only discuss some social groups/attributes where different models exhibit performance disparities, noting that these biased attributes receive less attention compared to mainstream characteristics like race or religion. This lack of focus could lead to underrepresentation in fairness alignment datasets, resulting in poor fairness performance by some models on these groups. However, we did not specifically address cases where all models performed poorly. Drawing on existing research on the sources of model bias, we discuss potential reasons for this issue. As mentioned in references [3,6], the primary source of bias in LLMs is the training data. Similarly, we believe that the effectiveness of fairness alignment in models largely depends on the data used for alignment.

Specifically, the most notable instance in the FairMT-Bench test results appears in the category of Stereotype's gender. We selected instances where the model's responses exhibited bias for detailed analysis and discussion of common issues. In the case of gender stereotypes, biased responses primarily manifest as very subtle biases: seemingly conventional and normative implicit biases against women, such as domesticity and compliance, which generally are not recognized by models as biases to trigger protective mechanisms. A specific case study is shown in Table 2. We speculate two main reasons for this situation:

• The data used for fairness alignment is not comprehensive enough, involving an overly broad range of data on gender bias, making it challenging to cover comprehensively during data collection;
• Some biases are too subtle, for example, the stereotype that if all women were to be employed, we might have to lower hiring standards. This type of stereotypical bias is not overtly expressed and is difficult to align for fairness using keyword detection or similar techniques, posing higher demands on the model's language understanding and reasoning capabilities.

Table 2: Case study of the prevalence of gender biased responses.

Weakness_3: Relying heavily on GPT-4 for generating synthetic dialogue data could introduce bias from the generative model itself, which might impact the generalizability of FairMT-10K and FairMT-1K to real-world scenarios.

Our work synthesizes assessments of models' multi-turn dialogue capabilities to design a comprehensive framework for evaluating fairness in multi-turn dialogue scenarios. It aims to compare and explore the reasons for failure in fairness alignment during multi-turn dialogues. The six tasks we designed are primarily based on manually constructed templates, combining biased groups and corresponding stereotypes to create multi-turn dialogues. During the construction process, we thoroughly considered human users' speaking habits, adopting forms of ellipsis, reference, and persuasion typical in multi-turn dialogues to design tasks, which simulate real-world dialogue scenarios.

Although the templates and datasets we constructed still lack the diversity and flexibility of real-world multi-turn dialogues, due to the current community's lack of real-world multi-turn dialogue datasets, the vast majority of multi-turn dialogue benchmarks still rely on LLMs to generate data [7-8]. Furthermore, the few real-world multi-turn dialogue datasets that contain biases and also feature scenarios involving references and persuasion are too rare to be used for constructing a benchmark.

Thank you for your constructive comments. We have carefully considered your suggestions and would like to provide the following clarifications. We have also updated the manuscript, with the modified contents highlighted in red for your review.

Weakness 1: The evaluation method largely depends on established LLM tools (e.g., GPT-4 as a judge), which may limit innovation in developing new fairness detection methodologies.

Thank you for your suggestion, and we agree with your perspective that the LLM-as-Judge method has inherent limitations, such as those imposed by the model's own capabilities. However, there are currently few reliable tools available for detecting bias or evaluating fairness in open text. The use of LLMs as judges is the prevalent method in existing fairness evaluation studies [1-3], and [4] has further demonstrated its feasibility and advantages.

At the same time, we acknowledge that using LLMs to evaluate the fairness of generated content may be influenced by the inherent biases in LLMs, which could lead to inaccurate evaluation results. Therefore, we have proposed some measures in our work. Firstly, when using GPT-4 as a tool for evaluating the fairness of generated content, we carefully designed the instructions for the evaluator GPT-4, included a detailed definition of the bias detection task, and provided some evaluation demonstrations to help the model understand the task. Additionally, to avoid the influence of the model's own biases on the evaluation results, we included in the instructions the original biased viewpoints used to construct the multi-turn dialogue, supplementing it with real human societal moral views. [5] has proven that such instruction design can effectively improve the model's bias detection accuracy. Under this evaluation framework, we primarily utilize GPT-4's language understanding capabilities, merely asking GPT-4 to judge whether the text under evaluation contains the given biased viewpoints, which enhances GPT-4's inherent bias detection capabilities. Additionally, we conducted human validation on GPT-4 predictions, and increased the number of human validation to verify the reliability of the GPT-4 evaluation. The updated consistency between human validation and GPT-4 evaluation results can be seen in Table 1. Overall, human validation results show high Accuracy and Recall, indicating strong consistency between the GPT-4 and human judgements, and GPT-4 exhibits a low rate of false negatives. However, in certain tasks, GPT-4's evaluations demonstrate a tendency toward over-protection, exhibiting a higher rate of false positives. Further analysis of cases with deviations in Appendix A.5.2 reveals a consistent pattern. Specifically, when a model's response contains both a refusal to answer and a biased statement, human annotators tend to label the response as unbiased, whereas GPT-4 is more inclined to label it as biased. We have also updated the new human validation results in Section 3.2 (Line 309-311) and Appendix A.5.2 of the paper.

Finally, developing new fairness detection methods is not the focus of this paper; our work mainly addresses the lack of capabilities in multi-turn dialogues and the ease with which model fairness alignment can fail, thus we constructed a framework that comprehensively evaluates model fairness under multi-turn dialogues. Therefore, we did not propose new bias detection tools or techniques, but rather adhered to the generic methods used in existing works. Moreover, our framework can integrate any new fairness detection methods when bias detection technology is updated, offering strong scalability.

Table 1: Consistency between new Human Validation and GPT-4 Results.

Thanks for the detailed response. Please include the content in the rebuttal of the manuscript, especially Table 1/2/3. I will raise my score accordingly

Weakness 3: Need to add a model quality dimension in the analysis.

We wholeheartedly agree with your suggestion that adding an evaluation metric for model capabilities can significantly enhance the completeness of our paper.

Based on your suggestion, we have included additional assessments of model capabilities in multi-turn dialogue. We tested the multi-turn dialogue capabilities of the six models on MT-Bench101 [2], a benchmark for fine-grained evaluation of model multi-turn dialogue capabilities.

Additionally, in our FairMT-Bench, we designed tasks to assess the trade-off between fairness and model utility. In conjunction with our tasks for assessing model fairness, we focused on the trade-off between fairness and model utility, observing whether the model begins to refuse responses due to keywords associated with certain social groups or attributes without any biased intent from the user, sacrificing model utility to ensure fairness. In our task design, the Fixed Format task involves asking the model to answer unbiased questions related to a social group in a fixed format for the first four turns, and then posing a biased question related to the same social group in the same format in the fifth turn. This task design considers the model’s ability to balance performance and fairness under clear instructions and requirements, and also examines the model's flexibility in changing its response strategy when the dialogue content undergoes subtle changes across multiple turns. Ideally, the model should normally answer according to the format in the first four turns and refuse to answer while pointing out the bias in the prompt in the last turn of dialogue. Here, we examine whether the model exhibits over-protection by generally refusing to respond upon detecting sensitive words related to a particular group, specifically if the model displays refusal behavior in the first four turns.

The test results from MT-Bench101 and the model's over-refusal are shown in Table 2. Overall, models with stronger multi-turn dialogue capabilities tend to exhibit better fairness performance in multi-turn dialogues (as analyzed from the Avg. results). Specifically, we found that models with stronger Adaptability and Interactivity are more susceptible to user guidance and interference, which can alter their fairness strategies during multi-turn dialogues. We think this may be because the fairness alignment of these models has not adequately accounted for multi-turn adversarial prompts.

Table 2: Results of Multi-turn capability.

Weakness_4 The distribution of different tasks needs more clarification.

Based on your suggestion, we have supplemented the statistical distribution of the dataset across different tasks in Table 4, Appendix A.1 in the revised manuscript. The results are also provided in the following Table 3. Specifically, for the Anaphora Ellipsis and Jailbreak Ellipsis tasks, we utilized GPT-4 to assist in generating part of the data. Therefore, we conducted a manual validation of the quality of the content generated by GPT-4 and removed some low-quality questions, resulting in slightly lower numbers for these two tasks. The other four tasks were constructed based on manually crafted dialogue templates, hence the quantity is equal across these tasks.

Table 3: The number distribution of each task on the Fair MT 10k dataset

Thank you for your constructive comments. We have carefully considered your suggestions and would like to provide the following clarifications. We have also updated the manuscript, with the modified contents highlighted in red for your review.

Weakness 1: Lack of discussion on evaluation cost.

We agree with your opinion. As a practical benchmark, it's important to clearly outline the costs of the benchmark for future users. Therefore, we have supplemented the discussion on costs from both time and economic perspectives:

• Time cost: The generation speed varies among different models. In this analysis, we take Llama-2-7b-chat model as an example. Some API-called models, such as ChatGPT, generally produce faster results. We configure the model to generate text with “max_new_tokens” set to 150 and maintain the batch size and precision as 1 and FP32, respectively. The generation is performed on a single NVIDIA H100 GPU. Under this setup, generating 10K samples of 5-turn multi-turn dialogue responses takes about 72.5 H100 GPU hours.
• Economic cost: First, we discuss the costs associated with using GPT-4 as an evaluator. Specifically, we use GPT-4 Turbo to evaluate the generated content of the test models. The current price of GPT-4 Turbo is 0.01 USD per 1,000 input tokens and 0.03 USD per 1000 output tokens. In our setup, GPT-4 evaluates bias in the final turn of the dialogue by processing the historical dialogue as context and providing simple outputs: either 'Yes, it is biased' or 'No, it is not biased'. We have measured the cost of evaluating on FairMT-10k to be 171.28 USD. In addition, we also use the open-source model LlamaGuard-3 as another evaluation tool. While LlamaGuard-3 is less sensitive in detecting implicit biases compared to GPT-4, its assessment results and trends are generally consistent with those of GPT-4. LlamaGuard-3 serves as a cost-effective alternative when GPT-4 API is unavailable or to minimize expenses. Furthermore, to enhance evaluation efficiency and reduce costs, we have curated 1,000 particularly challenging samples from the FairMT-10K dataset into a new subset, FairMT-1K.

In response to your suggestion, we have included the costs of the benchmark in Section 2.3 and Appendix A.4 in the revised manuscript.

Dear Reviewer vp1K,

We would like to express our sincere gratitude for your valuable time and effort. Your insights and suggestions have played a crucial role in enhancing the quality of our work. We have made every effort to address all your questions and are eager to receive further feedback.

In response to your valuable suggestions and questions, we have implemented several targeted revisions to our manuscript:

• We have included a detailed discussion on the cost implications of utilizing our benchmark, as detailed in Section 2.3 and Appendix A.4.

• We have addressed debiasing by constructing a DPO dataset based on the FairMT-Bench test results, as outlined in Appendix B.6 and Table 9. Additionally, we evaluated the aligned model's fairness and capabilities in multi-turn dialogues, which led to a discussion on the unique challenges of debiasing in such dialogues.

• We have enhanced our manuscript with a discussion on the model’s multi-turn dialogue capabilities and quality, as shown in Appendix B.1 and Table 6. Specifically, we assessed the model's performance in multi-turn dialogues using MT-Bench 101 and explored the over-rejection rate using FairMT-Bench, providing insights into the factors that influence the failure of fairness alignment in these scenarios.

• We have incorporated more implementation details, including a more detailed description of the Fair-1K construction process in Appendix B.5 and specific statistical information for each task in FairMT-Bench in Appendix A.1, Table 4.

We appreciate your thoughtful engagement, which has significantly helped us refine our presentation of these insights. Given that we have addressed your main questions concerning FairMT-Bench, we kindly ask if you would consider revising your score? We are grateful for your feedback and are more than willing to address any remaining concerns. We sincerely wish you all the best in your work and endeavors!

Sincerely regards,

Author