Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals

We thank the reviewer for the feedback and advice. As recommended by reviewer GEWc, we have introduced two subsections to the appendix including a detailed description of the causal estimation pipeline (G.1) and the evaluation pipeline of explanation methods (G.2).

Regarding Eq.2:

This is a smart take. Eq.2 defines an estimator for the ICaCE, which is the causal effect of a concept on the classifier prediction. The estimator is the difference between predictions of an example and its approximated counterfactual (before and after the treatment). Notice that if the expectation of the approximation error is zero, then the estimator is unbiased and measures accurately the ICaCE (see our proof in Appendix A).

We agree that if the outputs of the classifier are flat (or have a high uncertainty), then the difference is small (or has a small norm in case it is a vector). However, so is the real ICaCE - it is not a limitation or weakness of the estimator, but the natural behaviour of the causal effect (see Goyal et.al., 2020) For example, when the classifier is random and assigns the same values for every example, then the causal effect would be 0 (i.e., the concept has no effect on the prediction), the same as the estimator. Nevertheless, we agree that when the uncertainty of the model is high, the causal effect is low, and the estimator and the results might be noisy. For this reason, we conducted multiple experiments and explained various models of different sizes and architectures.

Regarding your questions:

1. Your description of the estimation of the causal effect is accurate. We use the term “causal model” as a notation for “the causal representation model optimizing Eq.6”. Given an example, we first extract the representations of this example using the causal model. We then also extract the representations of the candidate set (this is done once at training since the candidate set does not change). We then find a match using Eq.4. After finding the match, we estimate the causal effect using Eq.2. Aspiring by your questions, we will include a page in the appendix and thoroughly detail the matching, estimation, and evaluation pipelines.

2. According to the generative approach, we prompt an LLM to generate a counterfactual, and then use Eq.2 to measure the causal effect. When we write “causal model” we mean the causal representation model, which is used for matching (see the first question). Notice that we do use an LLM to generate counterfactuals for training the causal model. However, we do not generate any counterfactual at inference time when finding a match.

3. For CEBaB: We use GPT-3.5-turbo for generating all the counterfactuals, for both the generative approach (zero-shot and few-shot, Table 2) and for generating CFs for training the causal model (matching approach). For the new Stance Detection dataset: We use GPT-4 for constructing the new dataset (see section 5.1 and Appendix D), but as with CEBaB, we use GPT-3.5-turbo for the generative approach and for training the causal model that we benchmark with the new dataset. We will clarify this in the final version.

Dear reviewer, we would like to discuss other weaknesses that you find in our paper with you. We hope we can convince you that this work is important. Please see our summarized contributions:

1. While the idea of causal explanations being integral to model faithfulness has been discussed in prior works, our paper is the first to provide a theoretical framework that unifies the concepts of faithfulness and causal explanation. Specifically, we prove that CFs (which are a fundamental concept in causal inference) guarantee faithfulness, unlike other model explanation techniques, such as probing, concept removal, adversarial approaches, etc.

2. In line with recent advancements in the capabilities of LLMs, we are the first to show that LLM-generated CFs provide the SOTA NLP model explanation.

3. Moreover, we demonstrate empirically that utilizing multiple CFs enhances the quality of the causal estimator across various explanation methodologies. This empirical validation is the first of its kind and is both important and useful.

4. This is the first paper that thoroughly explores matching techniques for causal model explanation in NLP. Specifically, we enhance the matching paradigm originating in the causal literature such that it considers the rich representation of the text, and not only the observed concepts. This key contribution lays the groundwork for future research in causal representation learning from text.

5. Finally, recognizing the gap in benchmarks for causal model explanations, we construct a new interventional benchmark. This demonstration paves the way for future research to explore causal model explanations for a broader range of NLP problems and more complex setups.

References:

Goyal, Yash, et al. Explaining classifiers with causal concept effect (cace). arXiv preprint arXiv:1907.07165, 2019.

Thank you for your valuable feedback. We appreciate your openness to further discussions.

Regarding the cost of counterfactual generation:

We agree with this comment and we actually discuss this issue in section 3 and Appendix B. Regardless, this approach may be used in high-stakes applications where accuracy is crucial. For other cases, we develop our efficient matching method, which is much faster and computationally cheaper. We would also like to refer the reviewer to the fact that the costs of strong LLMs like GPT-4 are becoming constantly cheaper (e.g. the recent announcement by OpenAI, which cut these costs by a half).

Regarding ablations and the quality of the candidate set:

The performance of matching methods indeed depends on the quality of the candidate set. Notice that we conducted extensive ablation experiments to explore the quality of the matching method as a function of the candidate set quality. While these experiments are in the appendix, they are referred to from the main paper.

1. In the ablation study (Appendix C, referred to from pages 5 and 7), we evaluated different models across three types of candidate sets: (a) the standard candidate set (randomly sampled from the training set), (b) the standard set augmented by ground-truth CFs, and (c) the standard set augmented by misspecified CFs (designed to challenge the matching model). The findings demonstrate that our full objective is the most robust across these variations, showcasing the adaptability of our method. This experiment underscores a key aspect of our paper: causality is crucial for learning faithful explanations.

2. Using our novel dataset (section 5.1 and Appendix D), we explored scenarios where the candidate set consists of out-of-distribution texts from topics that substantially differ from those of the primary examples (e.g., abortion vs climate change). Here, our method proved to be the most effective among matching methods, indicating its robustness even in less ideal conditions.

3. The first row of Table 2 (main results) shows our method's performance when the candidate set is augmented by ground-truth CFs. In this setup, our matching method surpasses even the generative models, illustrating its exceptional effectiveness under optimal conditions.

Regarding niche tasks with unknown concepts:

This paper focuses on model interpretability via causal effect estimation (i.e., providing an accurate estimate of how a change of a given variable impacts the prediction of the model). In the causal effect estimation setup, we assume the existence of a causal graph (typically provided by domain experts), outlining the concepts and their interrelations. This assumption might require some effort of concept and causal graph specification, but it is a requirement for accurate causal estimation (see Shpitser and Pearl 2008 and Feder et.al. 2021 in the references comment). Moreover, as we theoretically show in the main theorem of the paper, a lack of information about the candidate concepts and the causal graph may result in an unfaithful explanation. Hence, we consider the predefined concepts a feature, rather than a limitation of our approach.

This aligns with prevalent practices in causal inference in NLP (see references comment). Additionally, pre-defined concepts (properties, attributes, aspects) are extensively used in the NLP model interpretability literature; for example, almost every probing, fairness, and bias detection paper assumes their existence (see references comment). We recognize the significance of this discussion, and we plan to include it in the final version of the paper.

Regarding open-ended tasks:

This is a great question. Our framework is not applicable to open-ended tasks (and generation tasks) since the causal effect estimators assume a classifier. We are not aware of any work that aims to address causal model explanations for non-classification tasks, and we consider this an important topic for future research.

Regarding nested counterfactuals:

If “nested counterfactuals” refers to an intervention on a concept that is a parent of a mediator (e.g., given A->B->C, a change in A causes a change in B that changes C), then unfortunately, CEBaB - which is the only causal-driven benchmark for explanation methods - does not support such structures. Notice, however, that this work is the first step for further research, and the ideas proposed by the reviewer indeed provide great directions for future work. We believe that this paper laid the foundation for this kind of research: we show theoretically and empirically that faithful explanations must be causal.

Moreover, we demonstrate that LLMs can be utilized for constructing benchmarks for causal explanation methods that support your ideas: complex causal graphs, many classes, and nested CFs.

Update:

Please see our general comment about updates to the manuscript. Specifically, we added subsections discussing the pre-defined concept requirement and provided examples of CFs generated by LLMs in the medical domain as well as for causal graphs beyond CEBaB. Furthermore, our additional experiments show that our causal matching method is equally effective in a completely unsupervised setup - where the labels are predicted using an LLM instead of human annotations.

Regarding your question (difference between Random match and Approx):

The difference between the two methods is that in the approx method, using concept classifiers trained on the training set, we first identify only candidates that share the same value of concepts as the examples and then randomly select a match from them. For example, if the intervention is Food (Positive -> Negative), and the remaining three values of the confounders are Ambiance (Negative), Noise (Negative), and Service (Positive) - We first find all candidates with values of Food (Negative), Ambiance (Negative), Noise (Negative), and Service (Positive), and then we randomly select a match from them. In Random Match, we do not perform the identification step, and only randomly select a match from the candidate set (in our example: texts with Food (Negative)).

References:

Shpitser Ilya and Pearl Judea. Complete identification methods for the causal hierarchy. Journal of Machine Learning Research, 9, 1941-1979., 2008.

Feder, Amir, et al. Causal inference in natural language processing: Estimation, prediction, interpretation and beyond. Transactions of the Association for Computational Linguistics, 2022, 10: 1138-1158.‏

Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart M. Shieber. Investigating gender bias in language models using causal mediation analysis. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 92650b2e92217715fe312e6fa7b90d82-Abstract.html.

Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. CausaLM: Causal model explanation through counterfactual language models. Computational Linguistics, 47(2):333–386, June 2021. doi:10.1162/coli a 00404. URL https://aclanthology.org/2021.cl-2.13.

Divyansh Kaushik, Eduard Hovy, and Zachary C. Lipton. Learning the Difference that Makes a Difference with Counterfactually-Augmented Data. arXiv:1909.12434 [cs, stat], February 2020. URL http://arxiv.org/abs/1909.12434. arXiv: 1909.12434.

Belinkov, Y. (2022). Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1), 207-219.‏

Elazar, Yanai, et al. Estimating the Causal Effect of Early ArXiving on Paper Acceptance. arXiv preprint arXiv:2306.13891, 2023.‏

Zhang, Justine; Mullainathan, Sendhil; Danescu-Niculescu-Mizil, Cristian. Quantifying the causal effects of conversational tendencies. Proceedings of the ACM on Human-Computer Interaction, 2020, 4.CSCW2: 1-24.‏

Wood-Doughty, Zach; Sphitser, Ilya; Dredze, Mark. Challenges of using text classifiers for causal inference. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing. NIH Public Access, 2018. p. 4586.‏

Elazar, Yanai, et al. Measuring Causal Effects of Data Statistics on Language Model'sFactual'Predictions. arXiv preprint arXiv:2207.14251, 2022.‏

Another answer to the "nested counterfactuals" concern:

First, our theorem is designed to be flexible and does not rely on a specific type of causal graph, thus, it holds true regardless of the complexity of the causal graph.

Second, consider a diamond structure causal graph consisting of the path A->B->T and the path A->C->T. A nested counterfactual case occurs with A as the treatment since it would influence the mediators B and C and the outcome variable T. Accordingly, both B and C should not be fixed, and this can be implemented by informing the LLM about the causal structure using its prompt, for example: “Generate a counterfactual by changing the value of the concept A. Notice that A impacts the mediator concepts B and C, which should not be necessarily fixed due to their potential change in response to A”. See the example in the comment below.

This exemplifies that even in the case of nested counterfactuals (when one concept causes a change to another concept) can be modelled with our methods.

The reason we did not explore such scenarios in our paper is that CEBaB does not support them. Yet, we strongly believe that modern strong LLMs like GPT-4 support the generation of such counterfactuals (See the example in the comment below.). Our paper underscores that LLMs can facilitate the creation of new interventional datasets, also ones that are more sophisticated than CEBaB, significantly advancing the research in causal explanations and benchmark construction.

Example Here is an example of a diamond structure causal graph and a counterfactual generated by GPT-4. The first path is: 'Party Affiliation' -> 'Sympathy toward Trump' ->'Text'. Simultaneously, the second path is 'Party Affiliation' -> 'Sympathy toward Nancy Pelosi' -> ‘Text’.

Notice that this is a simplified example, as there may be many more variables in the causal graph (for example, education could be a confounder as it might impact the affiliation and also the writing style which impacts the text). Nevertheless, the party affiliation should not directly impact the text, but only through mediators (in our example, sympathy toward Trump and Pelosi).

The text is taken from Feder et al. 2022:

"It was a masterful performance – but behind the sunny smile was the same old Trump: petty, angry, vindictive and deceptive. He refused to shake the hand of House Speaker Nancy Pelosi, a snub she returned in kind by ostentatiously ripping up her copy of the President’s speech at the conclusion of the address, in full view of the cameras."

We prompt GPT-4 to:

“Generate a counterfactual by changing the writer's affiliation to the republican party. Notice that the affiliation impacts the sympathy of the writer toward Trump and Pelosi, which should not necessarily be fixed due to their potential change in response to the affiliation.”

The following counterfactual was generated by GPT-4:

“It was a decisive and strong performance – behind President Trump's confident demeanor was the same resilient leader: focused, assertive, and forthright. He chose not to shake hands with House Speaker Nancy Pelosi, a decision she responded to with disrespect by theatrically tearing up her copy of the President’s speech at the conclusion of the address, in full view of the cameras.”

This outcome indicates that GPT-4 maintained the value of the mediators ('Sympathy toward Trump' and 'Sympathy toward Pelosi') consistent with the changed treatment concept ('Party Affiliation'). This example demonstrates the capability of LLMs in supporting the generation of nested counterfactuals.

Thank you for the feedback and discussion and for adjusting our score.

Regarding your questions:

On ablations:

Our experiments simulate scenarios where the matching set includes misspecified counterfactuals. These counterfactuals result from interventions on both the treated concept and an additional one (e.g., while the intervention was changing the service from positive to negative, the food aspect was also changed from positive to negative). These misspecified CFs. , which bias the estimation (increasing the error) are also highly plausible matches due to their significant semantic overlap with the query (e.g., discussing the same restaurant or menu). This scenario is even more challenging than just injecting noise into the candidate set because the bias we inject is within the most likely match. Notably, although the performance of our matching method declines in this scenario, this degradation is substantially less significant than the degradation in the performance of other ablative models.

On OOD results:

We did not check the support between the in-domain and out-domain sets, although we anticipate it to be very low. This is because the sets differ in the topic they discuss (in the stance detection task, given a topic and a text, the model needs to predict whether the text supports, opposes, or is neutral to this topic). For instance, if the query example is about abortion, the candidate set exclusively comprises texts on different topics (although other topics, like feminism, might have a small overlap with abortions). The topics in this setup are Abortion, Atheism, Climate Change, Feminism, and Hilary Clinton. Given the low support in these challenging OOD settings, the performance gap between our matching method and the baselines is less pronounced than in the CEBaB dataset. Nonetheless, our method continues to outperform the baseline, showcasing greater robustness.

On Table 2:

We acknowledge that ideal settings - where the matching set for a given example contains its counterfactual (CF) - are not realistic (yet, the likelihood of finding CF-similar candidates increases with the expansion of the candidate set). These ideal settings serve as an upper bound for our matching method. Nevertheless, we have rigorously tested our method in more challenging and realistic settings, such as with matching sets that include misspecified counterfactuals (“confusing CFs”), OOD, and in all of our experiments the matching sets are relatively limited in size. For instance, in the CEBaB dataset, the whole matching set comprises fewer than a thousand examples, and for a given intervention only about three hundred potentially align with a new treatment value. We believe these realistically reflect practical constraints. Furthermore, when the target domain is known, LLMs can be leveraged to augment the matching set with additional examples. This strategy enables a more comprehensive coverage of the target domain distribution.

‏Thank you for your valuable feedback. We appreciate your openness to further discussions.

Regarding the contribution of the work (first concern):

First, we use the term “approximated counterfactual explanation method” as an umbrella term for any explanation in our paper that is based on CFs. The Approx method introduced in CEBaB is just an example of such a method and is a naive adaptation of low-dimensional matching that is abundant in the literature on causal analysis. It only looks for examples with similar concept values, completely ignoring the semantics of the text, which is very different from our approaches.

Second, the theoretical analysis in Section 3.1 explains why any approximated CF explanation method is order-faithful, not only the naive Approx baseline from CEBaB, and underscores the importance of the causal approach to the explanations.

We would like to list the contributions of the paper, which we believe go well beyond the reviewer’s statement:

1. While the idea of causal explanations being integral to model faithfulness has been discussed in prior works, including CEBaB, our paper is the first to provide a theoretical framework that unifies the concepts of faithfulness and causal explanation. Specifically, we prove that CFs (which are a fundamental concept in causal inference) guarantee faithfulness, unlike other model explanation techniques, such as probing, concept removal, adversarial approaches, etc.

2. In line with recent advancements in the capabilities of LLMs, we are the first to show that LLM-generated CFs provide the SOTA NLP model explanation.

3. Moreover, we demonstrate empirically that utilizing multiple CFs enhances the quality of the causal estimator across various explanation methodologies. This empirical validation is the first of its kind and is both important and useful.

4. This is the first paper that thoroughly explores matching techniques for causal model explanation in NLP. Specifically, we enhance the matching paradigm originating in the causal literature such that it considers the rich representation of the text, and not only the observed concepts. This key contribution lays the groundwork for future research in causal representation learning from text.

5. Finally, recognizing the gap in benchmarks for causal model explanations, we construct a new interventional benchmark. This demonstration paves the way for future research to explore causal model explanations for a broader range of NLP problems and more complex setups.

In essence, our main contribution lies in theoretically and empirically establishing that faithful explanation methods must be causal and propose CFs as the central mechanism to achieve this. We believe that this insight is pivotal for the field of model interpretability.

Regarding the pre-defined concepts:

This paper focuses on model interpretability via causal effect estimation (i.e., providing an accurate estimation of how a change of a given variable impacts the prediction of the model). This differs from other model interpretability problems, such as finding concepts that may correlate with the prediction.

Accordingly, in the causal effect estimation setup, we assume the existence of a causal graph (typically provided by domain experts), outlining the concepts and their interrelations. This assumption might require some effort of concept and causal graph specification, but it is a requirement for accurate causal estimation (see Shpitser and Pearl 2008 and Feder et.al. 2021 in the references comment). Moreover, as we theoretically show in the main theorem of the paper, a lack of information about the candidate concepts and the causal graph may result in an unfaithful explanation. Hence, we consider the predefined concepts a feature, rather than a limitation of our approach.

This aligns with prevalent practices in causal inference in NLP (Feder et al., 2022, see references comment). Additionally, pre-defined concepts (properties, attributes, aspects) are extensively used in the NLP model interpretability literature; for example, almost every probing, fairness, and bias detection paper assumes their existence (see references comment). We recognize the importance of this discussion, and we plan to include it in the final version of the paper.

Here is an example of a diamond structure causal graph and a counterfactual generated by GPT-4. The first path is: 'Party Affiliation' -> 'Sympathy toward Trump' ->'Text'. Simultaneously, the second path is 'Party Affiliation' -> 'Sympathy toward Nancy Pelosi' -> ‘Text’.

Notice that this is a simplified example, as there may be many more variables in the causal graph (for example, education could be a confounder as it might impact the affiliation and also the writing style, which impacts the text). Nevertheless, the party affiliation should not directly impact the text but only through mediators (in our example, sympathy toward Trump and Pelosi).

The text is taken from Feder et al. 2022:

"It was a masterful performance – but behind the sunny smile was the same old Trump: petty, angry, vindictive and deceptive. He refused to shake the hand of House Speaker Nancy Pelosi, a snub she returned in kind by ostentatiously ripping up her copy of the President’s speech at the conclusion of the address, in full view of the cameras."

We prompt GPT-4 to:

“Generate a counterfactual by changing the writer's affiliation to the republican party. Notice that the affiliation impacts the sympathy of the writer toward Trump and Pelosi, which should not necessarily be fixed due to their potential change in response to the affiliation.”

The following counterfactual was generated by GPT-4:

“It was a decisive and strong performance – behind President Trump's confident demeanor was the same resilient leader: focused, assertive, and forthright. He chose not to shake hands with House Speaker Nancy Pelosi, a decision she responded to with disrespect by theatrically tearing up her copy of the President’s speech at the conclusion of the address, in full view of the cameras.”

This outcome indicates that GPT-4 maintained the value of the mediators ('Sympathy toward Trump' and 'Sympathy toward Pelosi') consistent with the changed treatment concept ('Party Affiliation'). This example demonstrates the capability of LLMs to support the generation of nested counterfactuals.

Thank you for your engagement in this discussion.

You are correct about the example - there was an error. We inadvertently pasted the counterfactual instead of the original example. We have now edited the comment and fixed this.

In addition, we acknowledge your feedback concerning the use of political examples. We plan to add an example from the medical domain to our comments and to a section in the appendix that discusses complex causal graphs with mediators.

Update:

Please see our general comment about updates to the manuscript. Specifically, we added subsections discussing the pre-defined concept requirement and provided examples of CFs generated by LLMs in the medical domain as well as for causal graphs beyond CEBaB. Furthermore, our additional experiments show that our causal matching method is equally effective in a completely unsupervised setup - where the labels are predicted using an LLM instead of human annotations.

Thank you for the insightful feedback and constructive comments.

Regarding using only the CEBaB dataset:

Our choice of CEBaB was driven by its unique status as the only non-synthetic interventional dataset tailored for benchmarking concept-level explanation methods in NLP. While automatic techniques for evaluating explanations exist and can be applied to non-interventional datasets, they predominantly assess correlation rather than causation (as discussed in our Introduction) and, therefore, are unsuitable for this paper.

Nevertheless, recognizing this limitation, we have not limited our investigation to CEBaB only. As detailed in subsection 5.1 and Appendix D, we introduced a new benchmark focusing on Stance Detection. This extension serves to validate our findings in diverse contexts, demonstrating the replicability of our approach.

Additionally, our work underscores a novel contribution in utilizing LLMs for generating Counterfactuals (CFs). This, as we demonstrated, can facilitate the creation of new interventional datasets, significantly advancing the research in causal explanations and benchmark construction. We plan to highlight these aspects more prominently in the revised manuscript.

Regarding pre-defined concepts:

This paper focuses on model interpretability via causal effect estimation (i.e., providing an accurate estimation of how a change of a given variable impacts the prediction of the model). This differs from other model interpretability problems, such as finding concepts that may correlate with the prediction.

Accordingly, in the causal effect estimation setup, we assume the existence of a causal graph (typically provided by domain experts), outlining the concepts and their interrelations. This assumption might require some effort of concept and causal graph specification, but it is a requirement for accurate causal estimation (see Shpitser and Pearl 2008 and Feder et.al. 2021 in the references comment). Moreover, as we theoretically show in the main theorem of the paper, a lack of information about the candidate concepts and the causal graph may result in an unfaithful explanation. Hence, we consider the predefined concepts a feature, rather than a limitation of our approach.

This aligns with prevalent practices in causal inference in NLP (Feder et al., 2022, see references comment). Additionally, pre-defined concepts (properties, attributes, aspects) are extensively used in the NLP model interpretability literature; for example, almost every probing, fairness, and bias detection paper assumes their existence (see references comment). We recognize the importance of this discussion, and we plan to include it in the final version of the paper.

Regarding your question:

As mentioned by the reviewer, we could generate the candidate set using LLMs, although this is not what we did in practice. We constructed the candidate sets as follows: For the CEBaB setup, the candidate set is randomly sampled from the training set. Notice that this candidate set is used only at inference time. For the novel Stance Detection setup, we rely on texts from the SemEval16 dataset but generate the entire training, dev, test, and candidate sets.

The only assumptions we make in this paper are about the pre-defined concepts and the causal graph, and the existence of training and dev sets. The CFs we use for training the models are generated by an LLM, making our matching approach more general compared to the alternative of using human-written CFs.

Update:

Please see our general comment about updates to the manuscript. Specifically, we added subsections discussing the pre-defined concept requirement and provided examples of CFs generated by LLMs in the medical domain as well as for causal graphs beyond CEBaB. Furthermore, our additional experiments show that our causal matching method is equally effective in a completely unsupervised setup - where the labels are predicted using an LLM instead of human annotations.