A Causal Lens for Evaluating Faithfulness Metrics

• The paper points out that many existing faithfulness metrics from [1] are flawed, which is useful extra evidence, though it is fundamentally unclear to me how this is going to lead to substantial later improvements.
• While the datasets used are systematic enough to gain insights, a big fear is that they are overly simplistic. The examples given are for very short chain-of-thought responses to basic questions. Moreover, most of the faithfulness metrics in [1] are quite premature and have been criticized in the literature [2, 3]. It makes sense to me therefore that CC-SHAP might perform well, but the performance here does not really shed insight into the performance of the metric on more subtle scenarios (nor is it clear how well the evaluation framework is able to handle real-world tasks such as medical question answering).
• These are my fundamental issues with the current setup, alongside relatively small/similar models being assessed. The faithfulness problem is due mostly to the fact that we do not have good ground truth of what faithful explanations are internally. This paper offers ground truths but in very simple settings.

[1] Measuring Faithfulness in Chain-of-Thought Reasoning, Lanham et. al., 2023

[2] Chain-of-Thought Unfaithfulness as Disguised Accuracy, Bentham et. al., 2024

[3] On Measuring Faithfulness or Self-consistency of Natural Language Explanations, Parcalabescu and Frank, 2023

My biggest concern is with the generation of synthetic explanations, and the assumption that one is correct and the other is incorrect. In particular, while the model is edited on the particular fact, it is unclear that the particular editing causes the model to use the "intended" reasoning path, or the model is actually using some very different reasoning paths. For example, in the Rihanna example, it could be that the model editing removes "Rihanna" entity from the "singer set", and hence results in the model predicting no. In this case, the "correct" explanation should be something like "No, because I do not find Rihanna in the singer set" (though this language may be highly unlikely to be generated by an LLM). How to carefully eliminate such possibilities is, in my opinion, the most crucial issue in establishing the soundness of the framework.

In addition, for a slightly related question, can we really assess the faithfulness of natural language explanations that are not generated by the models themselves (i.e., having low or almost-zero probability of being decoded), or in other words, is such evaluation fundamentally meaningful? In "traditional" interpretability, all the explanations, such as feature attribution, concept and counterfactual, are generated using well-defined algorithms, and hence the "meaningfulness" of the explanation is of little doubt. However, with synthetic explanations, I am less sure, so I would hope that the authors could convince me on this aspect.

Of a minor note, there are also alternative ways to generate and evaluate LLM-generated explanations, such as [1] and [2], that could be discussed in related work. Furthermore, there is even work that outright assert that LLMs cannot explain via natural language explantions [3].

[1] https://arxiv.org/abs/2310.11207

[2] https://aclanthology.org/2024.findings-acl.19.pdf

[3] https://arxiv.org/pdf/2405.04382

1. In Equation (5), the assumption is made that $\overline\epsilon_i$ is faithful to $\overline\theta$, while $\widetilde\epsilon_i$ is not. However, this is not guaranteed in the experiments. Though the authors discuss this in Section 5.2, it's unclear why perplexity can indicate faithfulness.

2. The model editing discussed in Section 2.2 involves modifying the internal weights of LLMs, which restricts the experiments to open-source LLMs, excluding closed-source models like GPT-4.

3. The model $\widetilde\theta$ is exclusively used to generate unfaithful explanations and is not incorporated in Equation (5). Since directly modifying $\overline\epsilon$ could also yield an unfaithful explanation related to $\overline\theta$, the necessity of using $\widetilde\theta$ is unclear.

4. The placement of figures throughout the paper is disorganized, which hurts readability.

1. The paper seems very applied to me with limited novelty. The authors expand an existing metric (called diagnosticity) to natural language explanations by arguing that random text cannot work as meaningful explanation (line 188..). However, this argument needs more backing/examples as random text can be considered as unfaithful explanation as done previously by Chan et al. 2022b.

2. Secondly, the authors introduce model editing as a way to generate pair of explanations (faithful and unfaithful), but this might be limiting the analysis as a given model editing method may not work perfectly (because the knowledge may be incorrectly learned), hence, model prediction may change, but the reason is incorrect. The authors argue to use "synthetic explanations" to handle this, but it is not clear if insights on synthetic explanations are generalisable to the real world. For e.g., are the syntetic explanations guranteed to not hallucinate?

1. The use of synthetic explanations may be limiting, as these pairs might not fully represent actual model-generated explanations. It would be helpful if the authors provided an analysis of how well synthetic explanations align with actual ones.
2. The focus on three specific tasks (fact-checking, analogy, object counting) may not generalize well to more complex contexts. Adding diverse tasks or discussing broader applicability would be helpful. Have the authors considered experimenting with other complex contexts?
3. Relying on diagnosticity as a faithfulness measure is overlooking other aspects of reasoning, like consistency and coherence. Including complementary metrics or discussing proposed framework’s limitations would be helpful.