Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought

1. My main concern with the proposed BCT is its potential negative impact on few-shot performance or, more broadly, in-context learning. While Appendix I shows some evidence for this, I'm not fully convinced, as Figure 9 only includes TruthfulQA results. I recommend including few-shot and few-shot CoT results on additional benchmarks, such as MMLU, to provide a more comprehensive assessment.
2. Section 5.2 demonstrates that BCT reduces coherent biased reasoning, highlighting a trade-off between CoT faithfulness and sensitivity to biases. This important finding warrants further discussion and analysis.
3. Some experimental design choices lack clear justification. For instance, the rationale behind measuring task and bias generalization simultaneously, as well as the selection of training data mixture proportions, could be better explained.

• Novelty: overall the proposed method is similar to existing literature on reducing bias in ML/NLP models with data augmentation or counterfactual augmentation, hence the novelty of the work is a bit lacking. The authors should provide more discussion on how this work is different from existing literature.

• For the generalization behavior, currently the main experiments are done on Sycophancy examples and generalization is shown on other bias types. It would be more useful to show if this trend holds or not (and to what extent) regardless of which bias type the model is being trained on. Some of the experiments are provided in Appendix E but the results are not very detailed. In Figure 5, why would multi-bias training perform on-par and sometimes even worse than single-bias? More discussion on this would be useful.

Minor:

• Many of the important results are included in the appendix, it would be more useful to re-organize the experiment section a bit and show all the major results in the main text. For example, the main results are only shown over one model: gpt-3.5-turbo-0613. The authors provided some additional experiments on LLaMa3-8B in the appendix, but it would be more useful to bring those experiments to the main text to show that the method generalizes across models. Also, is there a study on what model sizes are required for BCT to be effective?

While the presented approach to bias mitigation is interesting, several weaknesses limit its potential contribution.

• The paper lacks a clear theoretical explanation for the effectiveness of BCT, making it difficult to predict its applicability to different biases. While this is understandably a common challenge for LLMs, the resource-intensive nature of BCT, requiring numerous paraphrases for each bias type, raises concerns about its scalability and practicality. Additionally, the paper does not adequately address the potential challenges in generating high-quality paraphrases at scale.

• The inconsistent performance of BCT across different bias types raises questions about its reliability and generalizability. Furthermore, the observed degradation in instruction-following capabilities suggests potential unintended consequences, highlighting the need for a more thorough investigation of potential negative impacts. Perhaps its worthwhile to be more comprehensive in more instruction-following tasks.

• The experimental evaluation is limited in scope, focusing primarily on multiple-choice questions and bias reduction metrics. This narrow focus raises concerns about the generalizability of the findings to open-ended tasks and real-world applications. The paper's claim of successful generalization to unseen biases lacks a convincing analysis of the underlying mechanisms.

• The paper does not adequately position its contribution within the broader context of existing debiasing approaches. While BCT demonstrates some empirical success, its limitations and potential drawbacks raise questions about its overall significance and potential advantages over alternative methods.

• From Figure 2, I understand the main contribution of the paper is the BCT fine-tuning objective. However, there is no comprehensive explanation of what it entails, the assumptions, limitations of the proposed approach. Furthermore, Section 2 which is supposed to be the methodology part delves more around existing approaches, whilst leaving a single small paragraph for the explanation of the method.
• The experiments are limited to GPT-3.5T. Therefore, there is no demostration of how generalizable the proposed method is to LLMs in general.
• Beyond Figure 3, there is no other place in the experiments where the results of the proposed method in question answering tasks are clearly explained, with clear examples that show how this proposed method produces unbiased/biased answers.