On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models

Thank you for your thoughtful review of our paper. We are delighted that you recognize the importance of the issue of faithfulness, the soundness of our investigation, and the impact of our findings. We would like to address the weaknesses you pointed out and provide clarification on the questions raised.

W1: Number of benchmarks

We appreciate the reviewer’s suggestion to expand datasets/domains. This would surely enhance the paper’s original findings, though we do believe that this belongs in future work for two reasons. First, the objective of our study was to demonstrate that conventional strategies are not guaranteed to improve faithfulness, and second, we are interested in reasons why faithfulness is fundamentally hard to achieve. To do so, we emphasized a breadth of systematic investigation across many potential strategies for finetuning or in-context learning example selection, across reasoning vs non-reasoning datasets, in order to find failure cases and explanations as to why. We value the suggestion to explore other domains for additional insights in follow-up work.

W2: Explanations for success and failure of faithfulness

This is a very important point, to which we have dedicated our global response to. In fact, many explanations we originally had of why faithfulness is hard to achieve, were not prioritized correctly in the main text (see Appendix D). In general, faithfulness is not guaranteed. We thank the reviewer for pointing out the importance of these results for the main text.

Thank you for your thoughtful review of our paper. We are glad to hear that you believe our paper provides a strong basis for future exploration. We would like to address the weakness you pointed out and provide clarification.

W1/W4: Application of existing techniques and concrete next steps

We appreciate the reviewer’s viewpoint that we are utilizing existing techniques. The main purpose of our investigation is indeed to assess the abilities of existing intervention strategies, specifically on faithfulness. Ensuring the faithfulness of LLM-generated CoT reasoning is crucial for decision-makers, such as doctors, who rely on them to determine if, when, and how much to trust the recommendations made by these LLMs. It is therefore important to understand when and why these methods fail. The key contribution of our work is to explore if popular approaches that found success in modifying LLM outputs to improve properties like accuracy [1] and truthfulness [2] can improve the faithfulness of CoT reasoning generated by LLMs. Previous seminal works [3-6] have also explored whether common strategies are effective at solving certain problems across domains. The novelty and significance of these kinds of papers do not stem from a new approach or method but rather the significant insight that none of the popular approaches work for this problem, and we need a new set of approaches.

We are happy to release our coding framework (supplemental material) upon publication, which includes extensive pipelines that lay the groundwork for assessing faithfulness, using both OpenAI APIs and Hugging Face GPU implementations. Please note that this is a rapidly evolving field with many metrics / understandings of faithfulness and our work provides a strong datapoint upon which to build, and performing evaluations so extensively is not cheap. As a result, we uncover in-depth insights concerning the limitations of ICL (word mimicking, incentivization of late change in reasoning), and the fundamental uniqueness of trying to finetune for faithfulness (please see W3 below and our global response).

W2: On the particular faithfulness metric used

Measuring faithfulness of reasoning without having access to a black box is not straightforward, and hence, several works propose tests to evaluate faithfulness. Note that each test only evaluates an explanation of a particular property. We use the early answering test proposed by Lanham et. al. (2023) to measure faithfulness. The premise is that if reasoning is not post-hoc, there are fewer ways for it to be unfaithful than there are for reasoning which is post-hoc. While there are other possible faithfulness measures, they have their limitations as shown below (Table 1, Appendix).

Unlike the 'Adding Mistakes', and 'Paraphrasing' strategies, the 'Early Answering' strategy uses the generated CoT only from the model to measure faithfulness, thereby avoiding reliance on an external model/mechanism to evaluate faithfulness. However, we also looked at the 'Adding Mistakes' and 'Paraphrasing' strategies to measure faithfulness and found that both these measures are highly correlated with faithfulness from the 'Early Answering' strategy, shown in Fig 11 (Appendix). Our observation is also consistent with the finding reported in Lanham et. al. (2023).

Thank you for your comments. We will move Table 1 (above) and Fig 11 from the appendix to the main text, to prioritize the reviewer’s concerns (they remain in the appendix for now for identification purposes).

W3/Q1: Why is faithfulness fundamentally challenging?

This is a great point, and one that we did not highlight enough in the main text, though we have thought about it extensively. We kindly invite the reviewer to read our global response. The goal of faithful Chain of Thought (CoT) reasoning involves fundamentally altering the model to generate reasoning that is more consistent with its internal decision-making processes. This represents an intrinsic change in the model's behavior, rather than simply learning a new task. When trying to finetune based on faithful CoT examples, there is no guarantee that such examples remain faithful once model parameters are adjusted. In other words, supervised learning for faithfulness poses non-stationary/moving targets. We provide a carefully designed experiment to demonstrate empirical evidence of this. As it turns out, we made findings on the fundamental challenges of finetuning for faithfulness and the limitations of ICL, in Appendices D3 and D4, but failed to prioritize their relevance correctly for the main text. We therefore thank the reviewer sincerely for bringing this to our attention.

Experiment: we measure the faithfulness of the finetuned model’s CoT explanations with respect to both itself and the original model. This does not make sense in other domains with static datasets, but for faithfulness is crucially insightful, where there do not exist ground truth faithful (question, explanation) pairs. What our demonstrations strongly suggest is that pairs that are faithful w.r.t. the original model may no longer be guaranteed to be faithful w.r.t. a finetuned model, supporting our claim that faithfulness is fundamentally difficult to target using supervised learning.

Closing Remarks

Thank you again for your fruitful feedback! We would like to invite you to further discussion, in case your concerns are still not addressed. It would be great if you can let us know if you have any additional concerns, and we will be happy to respond. Should the insights satisfy your queries on why faithfulness is difficult to learn, we would strongly appreciate your vote of acceptance.

[1] Wei et al. Emergent abilities of large language models. TMLR, 2022.

[2] Li et al. Inference-Time Intervention: Eliciting Truthful Answers from a Language Model. NeurIPS, 2023.

[3] Goodfellow et al. Explaining and Harnessing Adversarial Examples. ICLR, 2015.

[4] Bolukbasi et al. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. NeurIPS, 2016.

[5] Adebayo et al. Sanity Checks for Saliency Maps. NeurIPS, 2018.

[6] Jain et al. Attention is Not Explanation. ACL, 2019.

Thank you for your thoughtful review of our paper, and for recognizing our main contribution that other intervention strategies may be required to achieve faithfulness, based on our investigation of existing techniques. We would like to address the weakness you pointed out and provide clarification.

W1/Q1: Insights on why faithfulness is difficult to learn, either in the form of mathematical theorems, or carefully designed experiments would be helpful

This is an important point and one that we did not highlight enough in the main text, though we have thought extensively about it. The goal of faithful Chain of Thought (CoT) reasoning involves fundamentally altering the model to generate reasoning that is more consistent with its internal decision-making processes. This represents an intrinsic change in the model's behavior, rather than simply learning a new task. This poses significant challenges when trying to learn faithful CoT reasoning, since there is no guarantee that any faithful examples used will remain faithful once model parameters are adjusted. In other words, learning faithfulness poses non-stationary/moving targets.

New experiments

We are happy to provide a carefully designed experiment to demonstrate empirical evidence of this. Please refer to our global response for full details. We measure the faithfulness of the finetuned model’s CoT explanations with respect to both itself and the original model. This does not make sense in other domains with static datasets, but for faithfulness is crucially insightful, where there do not exist ground truth faithful (question, explanation) pairs. Our findings demonstrate that once finetuning occurs, model internals shift, and what constitutes faithfulness changes. This is analogous to chasing a dynamic/moving target and poses fundamental challenges to standard static finetuning.

As it turns out, we made findings on the fundamental challenges of finetuning for faithfulness and the limitations of ICL, in Appendices D3 and D4, but failed to prioritize their relevance correctly for the main text. Therefore, we thank the reviewer sincerely for bringing this to our attention. We have dedicated a section in the main text to these findings.

Closing Remark

It would be great if you can let us know if you have any additional concerns, and we will be happy to respond. Should the insights satisfy your queries on why faithfulness is difficult to learn, we would strongly appreciate a vote of acceptance for our paper.

Thank you for your thoughtful review of our paper. We are happy that you recognize our paper’s contributions towards connecting many disparate concepts and filling a significant gap in evaluating for interventions. We would like to address the weaknesses you pointed out and provide clarification on the questions raised.

W1/Q1: Do you expect activation steering to be more or less effective for other models/architectures?

We appreciate the suggestion to trial AE on other models. We are actively collecting results for Gemma and appreciate your patience.

W2: Finetuning with most faithful explanations vs finetuning by length?

Thank you for the suggestion to finetune according to length, which would provide good insights into how CoT length affects faithfulness across different domains. We kindly refer the reviewer to Fig 11 (Appendix D.1, page 16), where we inspect test examples on which faithfulness remain the same/improved and observe that the average number of CoT reasoning steps used by each model generally increased (7/9 cases for ICL and 5/6 cases for FT). Models often invoked more granular CoT reasoning steps to improve faithfulness according to the early-answering metric. Mechanically, this is likely to increase the chances of a mismatch between intermediate and final answer probabilities (and thus the AOC). As such, we do not necessarily wish to optimize for faithfulness by learning shortcuts on the metric such as the above.

Additionally, please see our global response for evidence of why finetuning is a fundamentally difficult challenge in the context of faithfulness, which we deem particularly relevant to this discussion.

Q2: Will you be releasing code/data for how faithfulness was calculated in this particular case?

Absolutely! We are happy to release our coding framework upon publication, which includes extensive pipelines that lay the groundwork for assessing faithfulness, using both OpenAI APIs and Hugging Face GPU implementations.

W3/Q3: Do you expect your results to be consistent across different faithfulness metrics?

This is a great question. We conducted additional experiments to evaluate faithfulness on the AQuA dataset using strategies proposed by Lanham et. al (2023). We looked at the 'Adding Mistakes' and 'Paraphrasing' strategies to measure faithfulness and find that both these measures are highly correlated with faithfulness from the 'Early Answering' strategy, shown in Fig 12 in the pdf (Appendix D.2, page 17). Our observation is also consistent with the findings reported by Lanham et. al. (2023).

However, we do note that identifying fundamental challenges in applying interventions to one faithfulness measure (early answering) sheds much light on how optimizing for faithfulness can be severely limited. While extensive, unified analysis across further metrics would enhance the paper, we believe this belongs squarely in future work and does not detract significantly from our main illustrations regarding fundamental issues of faithfulness (namely, as per the global response, finetuning is a moving target, and ICL mimics word rather than learning intrinsic faithfulness, or learns other salient shortcuts such as increasing CoT granularity).

W4. The faithfulness based activation editing strategy was introduced, but the results on it were not included in the paper.

The results for activation editing strategy are presented in Section 4.2.3. Figure 10 shows how faithfulness and accuracy vary with the intervention hyper-parameters - number of heads $K$ and strength of intervention $\alpha$.

Thank you once again for your insightful suggestions and comments. We are actively incorporating the above explanations to enhance the quality of our paper. We believe that we have addressed all the concerns, barring additional activation editing results. If there is any aspect that you feel has not been fully resolved, we would be happy to provide further information. If you are satisfied with our response, we would truly appreciate your consideration in raising your evaluation score.

Thank you for your patience.

We appreciate the suggestion to trial Activation Editing strategy on other models. While we agree with the reviewer that effectiveness of activation steering can vary across models and attention mechanisms, Llama-3-8B-Instruct was a good starting point as it has proven to be effective in Li et al. (2023). We have performed activation editing (AE) experiments on Gemma-7B-IT and Mistral-7B-Instruct to make our results more robust as per the reviewer's requests. Please see page 22 and 23 of the updated manuscript. Gemma-7B-IT and Mistral-7B-Instruct are models of similar size as Llama-3-8B-Instruct but different architectures and attention mechanism. Gemma-7B-IT uses the well known self attention and Llama-3-8B uses grouped query attention (key and value heads are shared across a group of attention heads). In addition to grouped query attention, Mistral-7B-Instruct uses sliding window attention for efficient scaling to very long context windows.

For all three datasets (AQuA, LogiQA, TruthfulQA), we ablate over the number of intervened heads ($K$) and the intervention strength $\alpha$. As was the case with the Llama model, none of the intervention configuration leads to improvement of both accuracy and faithfulness. There is a significant trade-off in accuracy as faithfulness improves, and intervening beyond 8 attention heads often leads to illegible responses.

I'd like to point out that, while the inconsistency in hardness is the conclusion of your work, only considering one metric of faithfulness is perhaps premature to arrive at that conclusion. Your rebuttal pointed this out as well,

However, we do note that identifying fundamental challenges in applying interventions to one faithfulness measure (early answering) sheds much light on how optimizing for faithfulness can be severely limited.

I think it's less convincing if the definition of faithfulness is a moving target/not fully explored.

Thank you for your thoughtful review of our paper, and for recognizing the thoroughness of our experimental evaluations. We are glad that you found the experiments to be insightful (please see our global response for further insights we have attained for finetuning and in-context learning). We would like to address the weaknesses you pointed out and provide clarification on the questions raised.

W1: Novelty of our contribution

We appreciate the reviewer’s perspective on the novelty of the methodology. The key contribution of our work is to explore if popular approaches that found success in modifying LLM outputs to improve properties like accuracy [1] and truthfulness [2] can improve the faithfulness of CoT reasoning generated by LLMs. Previous seminal works [3-6] have also explored whether common strategies are effective at solving certain problems across domains. The novelty and significance of these kinds of papers do not stem from a new approach or method but rather the significant insight that none of the popular approaches work for this problem, and we need a new set of approaches that can address this problem. We illustrate that in-context learning, finetuning and activation editing offer limited success, and we also provide evidence as to why these methods are (fundamentally) limited in the context of faithfulness.

W2: AOC values are not calculated as a weighted sum as in Lanham et. al. (2023)

We would like to correct a potential misunderstanding here. In Lanham et. al. (2023), the “weighted sum” for AOC corresponds to a) grouping all CoTs by length, b) averaging across all instances in each group to create one profile per group, c) computing AOC for each group’s profile and d) computing a “weighted sum” across all groups. In other words, as in Lanham et. al. (2023): the AOC for each CoT length is weighted by the fraction of CoT samples having that length. This average the CoT graphs for each group length and then compute AOC for each, whereas our scores represent average AOC across all CoTs (instance-level). We hope this provides clarity! We are actively updating the text to clarify this.

W3: Can we compare Top-K faithfulness heads and Top-K truthfulness heads?

The work on truthfulness [2] is an experiment on a factual question answering dataset i.e.; TruthfulQA which contains questions like “What is the capital of France ?” for which there is a single correct answer in the options provided. Such tasks don’t benefit significantly from step by step reasoning. Top-K truthful heads in [2] correspond to the top attention heads whose representations are best predictors of an answer grounded in world knowledge. On the contrary, faithfulness is grounded in the model's inner workings. The exact same reasoning chain generated by two different models can have different values of faithfulness. Hence, top-K truthful heads and top-K faithful heads are different. Upon empirical evaluation, we observe the same. Following table shows the overlap between top-$K$ truthful and faithful heads for Llama-3-8B-Instruct model for varying $K$.

Q1: What is the sample size of the benchmark?

The sample size of each benchmark was 400 training examples from which to select responses (with 10 responses sampled per question in the case of 0.3 temperature sampling), and 100 test questions per dataset on which to evaluate faithfulness. This is motivated by a) high API costs and b) valuing breadth of evaluation across the several strategies that we introduce, rather than depth in test sample size (our evaluations show that mean test set faithfulness converged, on average, after around 60 to 80 of the 100 samples).

Q2: When selecting N ICL demonstrations, are we considering questions' similarities or just using faithfulness as the single index?

Faithfulness was considered as the index for selecting examples for ICL/FT, rather than similarity, since we effectively create a dataset of faithful question-explanation pairs from which the model can learn. Interestingly, this results in more faithfulness to the original model where the examples are initially drawn from (global response). There is no guarantee that finetuning on a model’s faithful or similarly related examples would result in improved faithfulness for the finetuned model. We appreciate the suggestion on similarity, though we believe it belongs squarely in future work.

Q3-5: Text corrections

We have updated the figure captions to reduce the requirement to browse around, thank you for the suggestion! We have also updated acronyms as suggested and fixed the mentioned typos, and will upload the new text soon once other edits are complete. We wholeheartedly appreciate the reviewer’s careful examination of our text.

Thank you once again for your insightful suggestions and comments. We are actively incorporating the above explanations to enhance the quality of our paper. We believe that we have addressed all the concerns. If there is any aspect that you feel has not been fully resolved, we would be happy to provide further information. If you are satisfied with our response, we would truly appreciate your vote of acceptance for our paper.

[1] Wei et al. Emergent abilities of large language models. TMLR, 2022.

[2] Li et al. Inference-Time Intervention: Eliciting Truthful Answers from a Language Model. NeurIPS, 2023.

[3] Goodfellow et al. Explaining and Harnessing Adversarial Examples. ICLR, 2015.

[4] Bolukbasi et al. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. NeurIPS, 2016.

[5] Adebayo et al. Sanity Checks for Saliency Maps. NeurIPS, 2018.

[6] Jain et al. Attention is Not Explanation. ACL, 2019.

Thank you for your thoughtful review of our paper and for recognizing the scope of our comprehensive experimental evaluations, aimed at uncovering the reasons behind limitations in existing intervention strategies for faithfulness. We would like to address the weaknesses you pointed out and provide clarification on the questions raised.

Q1: What is the formal definition of faithful (CoT) reasoning in LLMs?

Faithfulness represents how well a model’s expressed reasoning matches its internal reasoning. What constitutes internal reasoning for LLMs is a vastly complex topic, and as such can’t be reduced to a simple formal definition. It varies greatly across different faithfulness metrics (see lines 141-148). We will emphasize this aspect in our text to make it clearer to readers. Thank you pointing this out.

W1: The proposed solution is simply combining applied techniques in LLMs, which render the contribution incremental and straightforward

We appreciate the reviewer’s viewpoint that we are utilizing existing techniques. The main purpose of our investigation is indeed to assess the abilities of existing intervention strategies, specifically on faithfulness. Ensuring the faithfulness of LLM-generated CoT reasoning is crucial for decision-makers, such as doctors, who rely on them to determine if, when, and how much to trust the recommendations made by these LLMs. It is therefore important to understand when and why these methods fail. Previous seminal works [1-4] have also explored whether common strategies are effective at solving certain problems across domains. The novelty and significance of these kinds of papers do not stem from a new approach or method but rather the significant insight that none of the popular approaches work for this problem, and we need a new set of approaches that can address this problem.

The novelty in our contribution is the following. We illustrate that these methods achieve limited success, and we also provide significant evidence as to why these methods are (fundamentally) limited in the context of faithfulness. We kindly refer the reviewer to our global response for details. We in turn systematically evaluate across several novel example selection strategies for ICL/FT, finding that none are able to consistently improve faithfulness across domains. While conventional finetuning appears promising, faithfulness is a unique type of property that assesses a model’s internals and cannot be learnt as a supervised task.

W2: The aforementioned techniques have shown several limitations, in past works

Thank you for the feedback. We would like to clarify that fine-tuning and in-context learning are the widely used techniques to adapt pre-trained LLMs to specialized downstream tasks [5-10]. While we agree that there are existing limitations of the aforementioned techniques, it would be great if the reviewer could clarify which limitations would pose known challenges to faithful CoT reasoning.

W3: Several notions and techniques that this work builds upon, are not formally defined or described earlier in the paper, making it less accessible to a broader audience

We appreciate the reviewer’s feedback regarding the formal definition of CoT. We are actively resolving this in the updated text. If the reviewer could kindly provide specific examples of ambiguity we would be more than happy to address them and use them to strengthen our paper’s readability. Thank you for helping us improve our writing.

We are actively incorporating the above explanations to enhance the quality of our paper. We believe that we have addressed all the concerns. If there is any aspect that you feel has not been fully resolved, we would be happy to provide further information. If you are satisfied with our response, we would truly appreciate your consideration in raising your evaluation score.

References

[1] Goodfellow et al. Explaining and Harnessing Adversarial Examples. ICLR, 2015.

[2] Bolukbasi et al. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. NeurIPS, 2016.

[3] Adebayo et al. Sanity Checks for Saliency Maps. NeurIPS, 2018.

[4] Jain et al. Attention is Not Explanation. ACL, 2019.

[5] Dettmers et al. Qlora: Efficient finetuning of quantized llms. 2023.

[6] Hu et al. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.

[7] Jeong et al. Domain-specialized llm: Financial fine-tuning and utilization method using mistral 7b. In Journal of Intelligence and Information Systems, 2024.

[8] Kumar et al. Fine-tuning, quantization, and llms: Navigating unintended outcomes. arXiv, 2024.

[9] Rafailov et al. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS, 2023.

[10] Singh et al. Whispered tuning: Data privacy preservation in fine-tuning llms through differential privacy. Journal of Software Engineering and Applications, 2024.