Defining and extracting generalizable interaction primitives from DNNs

Q3: Insights into the interpretability of extracted interactions.

Q3: "Can the authors elaborate on whether the distinctive interactions extracted from more powerful models, such as BERT-large, result in enhanced interpretability?"

A3: A good suggestion. We believe that the interaction is a better perspective to explain the representation similarity between different AI models than other metrics. It is because different AI models usually have different structures, and their features are not directly aligned to each other. It is a significant challenge to quantify the similarity of two DNNs. For example, let $f_A$ and $f_B$ denote the features of two networks A and B. Then, we learn an model $g$ to align $f_B$ to $f_A$, via $g=argmin \vert\vert f_A-g(f_B)\vert\vert^2$. Either $2\vert\vert f_A-g(f_B) \vert\vert / (\vert\vert f_A\vert\vert +\vert\vert g(f_B)\vert\vert)$, or $cos(f_A,g(f_B))$ still has certain technical flaws in evaluating the distance/similarity of two features. It is because we cannot theoretically guarantee that the numerical classification utilities of $f_A$, $g(f_B)$, and $f_A-g(f_B)$ are exactly proportional to their feature strength $\vert\vert f_A\vert\vert$, $\vert\vert g(f_B)\vert\vert$, and $\vert\vert f_A-g(f_B)\vert\vert$. Similarly, there is no theoretical guarantee for the faithfulness of $cos(f_A,g(f_B))$.

In comparison, when we use interactions, [cite 4] have found that the network output can be faithfully decomposed into different interaction effects, no matter whether two DNNs have fully different architectures. Besides, because the interaction effect is directly responsible for network output, we do not face the problem that the feature strength is not the classification utility. Therefore, we can use the similarity of interactions as the similarity of inference logics of two DNNs.

Furthermore, we have conducted a new experiment to compare the interactions extract from two LLMs, i.e., LLaMA and Aquila-7B [cite 3]. As Figure 11 in Appendix N shows, a pair of LLMs usually share more ratio of interactions than other pairs of language models. It means that most LLMs may usually converge to the same set of potentially true interactions, as long as the LLM has a huge number of parameters and is well trained on extensive data. We believe that our finding may help people interpret the common learning direction of different LLMs.

[cite 3] BAAI. Aquila-7B, 2023. URL https://huggingface.co/BAAI/Aquila-7B.

[cite 4] Li, M., et al. Does a neural network really encode symbolic concept? In ICML, 2023.

Thank you for your insightful comments and suggestions. We would like to answer all the questions and clarify your concerns. We have carefully revised the paper in blue according to your suggestions.

Q1: Asking to conduct ablation studies.

Q1: "a more detailed discussion on the choice of the parameter $\alpha$" "An ablation study exploring how variations in $\alpha$ affect the outcomes"

A1: Thank you. We have followed your suggestions to conduct a new ablation study to illustrate the impact of $\alpha$. Specifically, we jointly extracted two sets of AND-OR interactions from the $\rm BERT_{BASE}$ and $\rm BERT_{LARGE}$ models, which were trained by [cite 1] and further finetuned by us for sentiment classification on the SST-2 dataset. We set the $\alpha$ values to [0, 0.2, 0.4, 0.6, 0.8, 1.0] for comparison. Figure 8 in the revised paper shows that as $\alpha$ increases, the sparsity of the extracted interactions increased and the generalization power of the extracted interactions gradually decreased. Please see Figure 8 in Appendix K for details of the new ablation study.

[cite 1] Devlin, J., et al. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.

Q2: Inconsistencies in performance Improvements.

Q2: "The observed discrepancies in performance improvements … warrant a deeper investigation and explanation."

A2: Thank you. This is a deep insight into the LLMs in feature representations. Since LLaMA and OPT-1.3B have much more parameters than BERT-Base and BERT-Large, it is often widely believed that these two LLMs can better converge to the true language knowledge in the training data. Although the following answer is far beyond the research in this paper, we are still glad to answer this question.

We can roughly understand this phenomenon by clarifying the following two phenomena. First, through extensive experiments, the authors have found that the learning of a DNN usually has two phases, i.e., the learning of new interactions and the forgotten of incorrectly learned interactions (this is our on-going research). In most cases, after the forgetting phase, the remained interactions of an LLM are usually shared by other LLMs. The high similarity of interactions between LLMs has also been observed in [cite 2].

Second, compared to LLMs, relatively small models are less powerful to remove all incorrectly learned interactions. For example, a simple model is usually less powerful to regress a complex function. The less powerful models (BERT-Base and BERT-Large) are not powerful enough to accurately encode the potentially complex interactions.

Therefore, small models are more likely to represent various incorrect interactions to approximate the true target of the task. This may partially explains the high difficulty of extracting common interactions from two BERT models, as well as why the proposed method shows more improvements in the generalization power of interactions on BERT models.

Experimental verification. To this end, we have further conducted a new experiment to compare the interactions extracted from two LLMs, i.e., LLaMA and Aquila-7B [cite 3]. As Figure 11 in Appendix N shows, two LLMs usually encode much more similar interactions than two not-so-large models. These experiments partially verify our hypothesis.

[cite 2] Shen, W., et al. Can the Inference Logic of Large Language Models be Disentangled into Symbolic Concepts? arXiv preprint arXiv:2304.01083, 2023.

[cite 3] BAAI. Aquila-7B, 2023. URL https://huggingface.co/BAAI/Aquila-7B.

Thanks a lot for all the valuable comments and suggestions. We would like to answer all the questions and clarify your concerns. Besides, we have carefully revised the paper in blue according to your suggestions.

Please let us know as soon as possible, if you have any further questions or concerns.

Q1: $\color{blue}{\textsf{Ask for the importance of generalizable interactions for AI interpretability.}}$

Q1: "why generalizable interactions are important for AI interpretability" "the generalizability is defined on different models of the same task, which does not align with the conventional understanding of DNN generalizability."

A1: Thank you. Unlike traditional studies on the generalization power of a DNN, our research is led by a different motivation, i.e., exploring the optimal way to explain inference logics of a DNN into symbolic interaction concepts. To this end, although the huge gap between symbolism and connectionism is well known, the counter-intuitive phenomenon of the sparsity of interactions in [cite 1] provides a new potential for the breakthrough of the seemingly impossible task. Specifically, previous studies [cite 1] [cite 2] have observed the sparsity of the interactions extracted in well-trained DNNs (in fact, this has been partially proven [cite 3] under three common conditions).

Therefore, our study of ensuring the interactions' generalization power may serve the last piece to prove/identify the symbolic/sparse interactions that faithfully explain a DNN. If we simultaneously ensure that (1) each DNN just encodes a small number of salient interactions, comparing with all $2^n$ potential combinations in the input variables, (2) a small number of salient interactions can universally match the DNN outputs on all $2^n$ randomly masked sample $\\{\mathbf{x}_S|S\subseteq N\\}$, and (3) most interactions are generalizable over different DNNs trained for the same task, then we can consider that we have faithfully defined interaction concepts encoded by a DNN.

The proof of a DNN’s encoding of symbolic interactions may help future analysis of generalization power of an AI model. First, theoretically, we can redefine the generalization power of a DNN based on the learned interactions encoded by the DNN in future work. In fact, the vanilla Harsanyi interaction has been used to explain the generalization power of a DNN [cite 4], in which low-order interactions have been found to have higher generalization power than high-order interactions. Other traditional studies mainly analyze generalization power in terms of the accuracy and the loss difference between training and testing samples. However, none of these metrics can reflect the intrinsic mechanism of the generalization power of a DNN. In comparison, the interaction concept provides us an alternative perspective to redefine the generalization of a DNN. Specifically, given multiple DNNs trained for the same task and an input sample, if an interaction can be extracted from all/most of these DNNs, then we consider it generalizable. In this way, we can identify a set of generalizable interactions and a set of non-generalizable interactions, which show much deeper insights into the generalization power of a DNN.

[cite 1] Li, M., et al. Does a neural network really encode symbolic concept? In ICML, 2023.

[cite 2] Ren, J., et al. Defining and quantifying the emergence of sparse concepts in dnns. In CVPR, 2023.

[cite 3] Ren, Q., et al. Where we have arrived in proving the emergence of sparse symbolic concepts in ai models. arXiv preprint arXiv:2305.01939, 2023.

[cite 4] Zhou, H., et al. Concept-Level Explanation for the Generalization of a DNN. arXiv preprint arXiv:2302.13091, 2023.

Q2: Asking for the proof of Theorem 3.

Q2: "if detailed derivations of Equation 11 were provided" "The proofs in the paper rely on two papers currently under review ([1,2] See appendix C). "

A2: Thank you very much for your careful review. Please see the revised derivations of the Equation (11) in Appendix D.

First, the proposed loss function of learning generalizable interactions does not rely on the proof of two papers [cite 3] [cite 5]. We can still extract generalizable interactions without Theorem 3.

Second, the sparsity of interactions has been widely observed in many studies [cite 1] [cite 2], so that we actually do not need the two under-review papers [cite 3] [cite 5] to support the design of our algorithm. To this end, we are sorry that we use the inaccurate statements that mislead the understanding of Theorem 3. We have revised the paper in Theorem 3 and Proposition 1.

A2: Besides, the proposed universal-matching theorem on AND interactions has been proved in an CVPR paper [cite 2], as $\forall T\subseteq N, v(\mathbf{x}\_T)=\sum_{S\subseteq T}I\_{\text{\rm and}}(S|\mathbf{x})$. Then, we can prove the universal-matching theorem on both AND interactions and OR interactions (shown as the revised Theorem 3) in Appendix C. Then, according to the finding that well-trained DNNs tend to encode sparse interactions in [cite 1], we can also obtain the proposition $\forall T\subseteq N, v(\mathbf{x}\_T) \approx v(\mathbf{x}\_\emptyset) + \sum\nolimits_{\emptyset \neq S\subseteq T: S\in \Omega^{\text{\rm and}}}I\_{\text{\rm and}}(S|\mathbf{x}\_T) + \sum\nolimits_{S\cap T \neq \emptyset:S\in \Omega^{\text{\rm or}}}I_{\text{\rm or}}(S|\mathbf{x}_T)$. Given the findings in [cite 1], the proof of the sparsity of interactions in [cite 3] is no long so important to prove our conclusion.

[cite 1] Li, M., et al. Does a neural network really encode symbolic concept? In ICML, 2023.

[cite 2] Ren, J., et al. Defining and quantifying the emergence of sparse concepts in dnns. In CVPR, 2023.

[cite 3] Ren, Q., et al. Where we have arrived in proving the emergence of sparse symbolic concepts in ai models. arXiv preprint arXiv:2305.01939, 2023.

[cite 5] Li, M., et al. Technical note: Defining and quantifying and-or interactions for faithful and concise explanation of dnns, 2023. arXiv preprint arXiv:2304.13312, 2023.

Q3: "it lacks a discussion on the existence of generalizable interactions"

A3: Thank you. Theoretically, there is no guarantee that generalizable interactions necessarily exist. However, empirically, [cite 1] has observed that for many different tasks (e.g., image classification on the MNIST dataset and image classification on the CelebA-eyeglasses datasets), different DNNs trained for the same task all encode similar interactions. This is the motivation of this study, and we decide to go beyond empirical comparisons and formally design a loss function to learn generalizable interactions.

In addition, we have conducted more experiments to show the generalization interactions between more DNNs (i.e., between the LLaMA and Aquila-7B [cite 6]). Please see Appendix L for details.

Although we cannot strictly prove that such generalization power of interactions is a generic property for all tasks, experiments show that most DNNs exhibit interactions with considerable generalization power. We believe this will provide new insights into the representation quality of a DNN.

[cite 1] Li, M., et al. Does a neural network really encode symbolic concept? In ICML, 2023.

[cite 6] BAAI. Aquila-7B, 2023. URL https://huggingface.co/BAAI/Aquila-7B.

Q4: Discussion on "the method proposed in the paper is derived heuristically using Occam's Razor principle"

Q4: "The paper does not satisfactorily address why this approach would yield generalizable interactions."

A4: Thank you, but the generalization power of interactions is not guaranteed by the Occam’s Razor principle, although we do use this principle in Equation (6). Instead, we design a specific loss $\Vert\rm{rowmax}({I}\_{\text{\rm and}})\Vert_1 +\Vert\rm{rowmax}({I}\_{\text{\rm or}})\Vert_1$ in Equation (5), which exclusively penalizes non-generalizable interactions. Here, let us be given $m$ interactions on the same subset $S$, $I_{\rm and}^{(1)}(S|x), I_{\rm and}^{(2)}(S|x),...,I_{\rm and}^{(m)}(S|x)$, extracted from $m$ DNNs. We consider the interaction with the maximum absolute value $max_i \vert I_{\rm and}^{(i)}(S|x)\vert$ as the outlier effect that is exclusively contained by a single DNN. Thus, this loss only penalizes the outlier effect to boost the generalization power. For example, given $I^{(1)}(S|x)$ and $I^{(2)}(S|x)$ for the same set of input variables $S\subseteq N$ extracted from two DNNs, if this interaction is non-generalizable, it usually means that the interactions extracted from two DNNs have significantly different effects on the model output. In this way, the loss in Equation (5) directly pushes the most salient interaction towards the other interactions, thereby making interactions in different DNNs similar to each other.

We have revised the paper to clarify this in the paragraph under Equation (6).

Thank you very much for your comments, which help us much improve the paper in the rebuttal phase. It seems that most concerns (Q3, Q4, Q5, Q6, Q8) mainly focused on the clarity of the technical details. We have carefully revised the paper in blue according to your suggestions on the paper-writing issues. We have also carefully answered all other questions, and have conducted new experiments as you requested.

Please let us know as soon as possible, if you have any further questions or concerns.

Q1: Discussion on the recommended paper in ICCV 2023.

Q1: "refer to other work in examining shared features between networks"

A1: Thank you. This paper is interesting. Unlike our objective of faithfully explaining DNNs into symbolic interactions, that paper identifies units (convolutional kernels) in different DNNs that share similar concepts. Beyond the analysis of feature maps of different units, our paper discovers that different DNNs trained for the same task tend to learn similar sets of interactions, as long as the different neural networks are sufficiently optimized, no matter whether these DNNs have different architectures and parameters. For example, as Figure 4 shows, two completely different LLMs surprisingly encode two similar sets of sparse interactions (phrases). Therefore, our attempt of extracting common interactions shared by different DNNs provides a new direction of pursuing the essential explanation of a DNN’s feature representations.

Nevertheless, we have cited and discussed this paper in Appendix A in the revised paper.

Q2: $\color{blue}{\textsf{Asking about the evidence for the faithfulness of the sparsity and universal-matching.}}$

Q2: Concerning the paper's claim that "The sparsity and universal-matching property of interactions …to faithfully explain DNNs with interactions"."the methods are considered over only tiny subsets of just 8 or 10 features …cannot be claimed to be faithful to 'ResNet' or 'BERT Large' ", "The claim of faithfulness is not backed up empirically."

A2: It is a very good question. Let us clarify both in the theoretical proofs and in experiments for the faithfulness of the sparsity and universal-matching.

Theoretically, the faithfulness means that given an input with $n$ variables, we must prove that (1) the DNN only encodes a small number of salient interactions $\Omega$, comparing with all $2^n$ potential combinations of the input variables, i.e., $\vert\Omega\vert \ll 2^n$, and that (2) the network outputs on all $2^n$ randomly masked samples $\\{\mathbf{x}_S|S\subseteq N\\}$ can be well matched by a few interactions in $\Omega=\\{S\subseteq N: \vert I(S|x) \vert > \tau\\}$, $\Omega$ is defined in the eighth paragraph in Section 2.1 in the revised paper. These two terms have been proved in Theorem 1, Theorem 3 and Proposition 1 in the revised paper.

Then, in practice, considering the $2^n$ computational complexity, we have followed settings in [cite 1] to extract interactions between a set of randomly selected input variables $N=\\{1,2,…,t\\}, (t<n)$ (other unselected $(n-t)$ input variables will remain unmasked, leaving the original state unchanged). In this case, faithfulness does not mean that our interactions explain all the inference logic encoded between all $n$ input variables for a given sample $x$ in the pre-trained DNN. Instead, it only means that the extracted interactions can also accurately match the inference logic encoded between the selected $t$ input variables for a given sample $x$ in the DNN. We have clarified this in Appendix G in the revised paper.

Experimental verification. In addition, we have further conducted a new experiment to verify the faithfulness when we explain interactions between all input variables. To this end, for the sentiment classification task on the SST-2 dataset in ${\rm BERT_{BASE}}$, we selected sentences containing 15 tokens to verify the faithfulness of the sparsity in Proposition 1 and the universal-matching property in Theorem 3. All tokens are selected as input variables. We focused on the matching errors $\epsilon_T=v^{\rm approx}(x_T) - v(x_T), v^{\rm approx}(x_T):=v(x_{\emptyset})+ \sum_{\emptyset\ne S\subseteq T : S\in \Omega^{\rm and}}I_{\text{\rm and}}(S|x_T) + \sum_{ S\cap T \neq \emptyset: S\in \Omega^{\text{\rm or}}}I_{\rm or}(S|x_T)$ for all masked samples  $x_T, \forall T\subseteq N$. Specifically, we observed whether the real network output on the masked sample $v(x_T ), \forall T\subseteq N$ can be well approximated by interactions. We have verified that the extracted salient interactions in $\Omega$ faithfully explain the model output, i.e., $\forall T\subseteq N, v(x_T) \approx v(x_{\emptyset})+ \sum_{\emptyset\ne S\subseteq T : S\in \Omega^{\rm and}}I_{\text{\rm and}}(S|x_T) + \sum_{ S\cap T \neq \emptyset: S\in \Omega^{\text{\rm or}}}I_{\rm or}(S|x_T)$. Please see Appendix G  in the revised paper for more details.

A2: [cite 1] Li, M., et al. Does a neural network really encode symbolic concept? In ICML, 2023.

Q3: $\color{blue}{\textsf{About paper writing.}}$

Q3: "how continuous neural network features are discretized for the and-or analysis"

A3: Thank you. We have followed your suggestion to clarify how to define discretized interactions for continuous input variables in blue in the footnote [3]. We did not discretize the continuous values of each input variable. Instead, we simply followed [cite 1] to mask each input variable $i\in N\setminus S$ by defining and using baseline values (see Appendix C for the definition of the baseline values in the revised paper). For example, for the image dataset, we used zero as the baseline value to represent the masked state of the input variable in $i\in N\setminus S$ (see Appendix M.2 for experimental details in the revised paper). In this way, for each input sample with $n$ input variables, there are a total of $2^n$ different masked states $\\{x_S\\}$, as described in the tenth paragraph of Section 2.1. Therefore, the masked and unmasked states are two discretized states for each input variable, so that we can define the discretized interactions $I_{\rm and}$ and $I_{\rm or}$ based on the masked samples in Equations (1) and (2).

[cite 1] Li, M., et al. Does a neural network really encode symbolic concept? In ICML, 2023.

Q4: $\color{blue}{\textsf{About paper writing. Ask for detailed explanation for Equation 6.}}$

Q4: "How the search to optimize the discrete objective in Eq 6 is not discussed, and the level of sparsity" "the number of cohorts found in a typical solution is not revealed in concrete terms" "with both concrete examples of how the interaction matrices are formatted … in a specific small example" "a real-data example of how the method operates on a single example could be shown in detail"

A4: Thank you. Please see answers to Q3 for the concerns on the discretization problem. Then, let us answer other concerns as follows.

$\bullet$ The optimization of the loss function in Equation (6) is simply based on the gradient descent method. This is because the parameter $\gamma_{T}\in R$ to be optimized is continuous, rather than discrete. As mentioned in the first paragraph of Section 2.3, we learned the optimal values of $\gamma_{T}, T\subseteq N$ to decompose the model output $v(x_T)$ into two terms $v_{\rm and}(x_T)$ and $v_{\rm or}(x_T)$. The loss was designed to boost the sparsity of interactions, which ensured the sparsity of interactions. That is, such optimal values of $\gamma_{T}$ are learned to make most interactions almost zero effect, $I_{\rm and}(T|x) \approx 0$ and $I_{\rm or}(T|x) \approx 0$.

$\bullet$ The number of interactions been optimized. During the optimization process, the loss in Equation (6) is applied to all $2^n$ interactions. After the learning of parameters $\\{ \gamma_T | T\subseteq N\\}$, we compute the $2^n$ interactions based on Equations (1) and (2). Most of these interactions have almost zero effect. We select interactions which are greater than the threshold $\tau$ as salient interactions. The final number of salient interactions is determined by the threshold $\tau$. Please see the eighth paragraph of Section 2.1 for details.

Besides, we have also conducted experiments to show the number of salient interactions extracted by optimizing the loss in Equation (6). For example, Figure 2 shows that Equation (6) extracts 2915 salient interactions ($\approx$ 0.71%)  from the $(2\cdot2^{10})\times200$ AND-OR interactions extracted from a total of 200 samples in the ${\rm BERT_{BASE}}$ model, when we set the threshold $\tau=1/20\cdot\max_S\vert I(S|x)\vert$.

$\bullet$ The interaction matrices in Equation (6) are formulated in the first sentence below Equation (5).

Let us use a concrete example to illustrate how the interaction matrices are formatted in a toy example. Let us consider two pre-trained DNNs $v^{(1)}$, $v^{(2)}$ and an input sample $x$ with $N=\\{1,2\\}$ variables, and take the AND interaction matrix ${I}\_{\rm and}$ in Equation (6) as an example (the OR interaction matrix can be obtained similarly). First, we get all $2^2$ AND interactions extracted from the $i$-th model $\mathbf{I}^{(i)}\_{\rm and}=[ I\_{\rm and}(T_1|x), I\_{\rm and}(T_2|x), I\_{\rm and}(T_3|x), I\_{\rm and}(T_4|x)]^\intercal \in \mathbb{R}^{2^2}$, according to the description of Equation (4). Here, each interaction value $I_{\rm and}(T_k|x) \in \mathbb{R}, T_k \subseteq N$ denotes the interaction value of each masked sample $x_{T_k}$ and is computed by Equation (1). Second, the AND interaction matrix ${I}\_{\rm and}= \left[ \mathbf{I}^{(1)}\_{\rm and}, \mathbf{I}^{(2)}_{\rm and} \right] \in \mathbb{R}^{2^2\times 2}$ denotes the interaction values corresponding to the $2^2$ masked samples for each of the two models.

Thank you for the detailed example.

Providing such an example in a concrete case brings the paper in right direction because it makes the paper more accessible and more likely to be reproducible. However, my concerns about the clarity of the main paper remain. It would be helpful if the main paper included a clearer statement of both the method (more succinctly than your new appendix) and, especially if the main paper provided more of a discussion situating the new approach within the context of a broader range of related work.

For example: once the masking approach of the paper is clarified, the paper should also clarify how it differs from, or how it addresses limitations of previous methods that use masking to explain the behavior of a network (these include Petsiuk 2019, Fong 2017, Dabkowski 2017, Hanjie Chen 2020, and recently Johannes Linder 2022). What problems with those previous research works are addressed by this new paper's methods? The paper would benefit from a much more detailed discussion of other papers that have taken a related approach, and how they relate to the current work.

If rewritten with both (1) this clarity of the current method and (2) and its relation to the broader research context, the paper would be stronger, but my recommendation is that the paper still needs a major revision to reach this level of clarity. It would best be done as a major revision for another venue. (Remaining a 5)

Thank you for your feedback. We are glad to answer your questions. However, if you still have new concerns, we have no time to answer your new feedback before the deadline.

Compared to traditional masking methods, our approach is quite different. Instead, our approach aims to obtain an essential representation of the model's inference logic. We try to mimic the final output of the neural network with a sparse, small number of interactions. In contrast, these traditional masking methods have fully different tasks and different motivations.