Factor Graph-based Interpretable Neural Networks

We appreciate your valuable feedback, which are insightful and can improve the quality of our work to a great extent. Definitely, you point out some issues in our paper, giving us inspiration on how to revise the manuscript. In response to your comments, we have provided detailed answers and revised the manuscript accordingly.

Weaknesses 1:

We have made additions in the Rebuttal Revision of the paper (Figure 3). The final output include $\hat{c}_{re}$ and prediction. $\hat{c}_re$ is the corrected explanation.

Weaknesses 2:

Following your comments, we revised the manuscript to provide more experimental results about accuracy comparison. We have made additions in the Rebuttal Revision of the paper (Line 1242-1330 and Line 438-465).

Weaknesses 3:

We appreciate your constructive suggestions. Yes, the left bar chart represent the initial interpretation, and the right bar chart represent the interpretation combined with the factor graph. We revised the Rebuttal Revision of the paper (Line 506).

Weaknesses 4:

We apologize for these typos. We have made additions in the Rebuttal Revision of the paper (Line 087 and Line 312).

Weaknesses 1:

This is indeed a limitation and thank you for your valuable comments. In fact, not only factor graphs require predefined logic rules, but all methods based on external knowledge integration inevitably require predefined knowledge. Therefore, how to automatically extract knowledge from the external environment is a separate and important problem. We have already discussed this limitation in the conclusion. (Line 537-539)

Weaknesses 2:

In theory, the computational overhead of AGAIN's intervention is indeed high. However, in practice, the number of interventions is usually no more than 100, because erroneous concepts are typically few in number. For example, in the CUB dataset, the number of incorrect concepts is usually only 1-10 [1]. Intervention operations are performed on a small number of concepts, as these erroneous concepts are localized during the detection phase. Even so, reducing the computational overhead of interventions is a valuable topic for discussion. Additionally, we have added a set of comparison experiments (Line 1173-1208) to address Question 3.

Questions 1:

Thank you for your comment. Indeed, the process of injecting black-box perturbations into input data is by nature adding input noise.

Questions 2:

Following your comments, we revised the Rebuttal Revision to provide more details about computational efficiency analysis (Line 1143-1171).

Questions 3:

These strategies can reduce overhead in theory, but their practical payoff is not as significant. For further details, please refer to the experimental analysis in the Rebuttal Revision (Line 1173-1208).

[1] Understanding and enhancing robustness of concept-based models. AAAI.

Weaknesses Major Issues 1:

Thank you for your comment. Here, we’d like to highlight:

(1) In this paper, we focus on generating comprehensible explanations under unknown perturbations instead of injecting knowledge into the model. To address this problem, we require an effective way to inject external knowledge. We demonstrate in a supplemental ablation analysis (Line 366-375) that knowledge injected through factor graphs is more helpful in generating comprehensible explanations compared to other knowledge integration methods. After injecting external knowledge, we design an error detection mechanism and a concept intervention strategy to identify and correct wrong concept activations, thereby improving the comprehensibility of explanations.

(2)Although AGAIN also injects external knowledge, it has a distinctly different research goal and research question compared to [1-6]. Specifically, [1-4] focus on constraining the model to output correct predictions, rather than ensuring the model produces correct explanations under adversarial attacks. As a result, the model may still produce incorrect explanations even when it predicts correctly. [5] highlights that external knowledge can enhance interpretability, but it does not explain how the model ensures interpretability when explanations are disturbed by unknown perturbations. Similarly, LEN [6] learns rules between concepts and categories to correct predicted categories, but it does not address the correction of concepts. Therefore, we believe that none of these studies tackle the issue discussed in our paper.

(3)You raised concerns about the sufficiency of the comparison methods, which suggests that other readers may be similarly confused. Therefore, we have added experiments to compare the performance differences between AGAIN and the methods listed by you, focusing on knowledge integration. This experiment was included in the Rebuttal Revision of the paper (Line 366-375). The results show that the knowledge injected by the existing methods relies solely on forward reasoning to correct predictions and cannot reverse the conditional probabilities of the explanations based on those predictions. As a result, these methods lack the ability to correct explanations. In this regard, factor graphs offer an irreplaceable advantage.

Weaknesses Major Issues 2:

We appreciate your constructive suggestions. Based on your suggestions, we have added concept-based models to the related work and preliminaries sections. These additions can be found in the Rebuttal Revision of the paper (Line 117-121 and Line 140-145).

Weaknesses Minor Issues:

We have made additions in the Rebuttal Revision of the paper (Line 056, Line 082, Line 083, and Line 103).

Question 1:

We added experiments to compare the performance differences between AGAIN and knowledge-based integration methods. These experiment were included in the Rebuttal Revision of the paper (Line 366-375).

Question 2:

We plan to extend AGAIN to overcome the above limitations from two aspects:

(1) Domain knowledge changes: we will design modules to automatically mine logical rules, such as automatically constructing knowledge graphs, mining logical rules from knowledge graphs, and adaptively learning external knowledge from different scenarios.

(2) Correct prediction categories: we will enhance the logical association between concepts and concepts to reduce the sensitivity of external knowledge to prediction categories.

[1] DeepProbLog: Neural Probabilistic Logic Programming. NeurIPS.

[2] Deep Learning with Logical Constraints. IJCAI.

[3] Exploiting multi-object relationships for detecting adversarial attacks in complex scenes.CVPR.

[4]Domain knowledge alleviates adversarial attacks in multi-label classifiers.TPAMI.

[5] Improving deep learning with prior knowledge and cognitive models: A survey on enhancing interpretability, adversarial robustness and zero-shot learning.Cognitive Systems Research.

[6] Ciravegna, Gabriele, et al. "Logic explained networks. AI.

Thank you for your prompt replies. The comments you have raised help us to improve the quality of our paper. Based on your comments, we have revised the Rebuttal Revision and provided detailed responses below.

1.

(1)We did not limit our comparison to LSM alone; we also evaluated standard CBM metrics, including E-ACC and P-ACC. These metrics are widely adopted in studies related to CBMs [1-3]. Due to space constraints, descriptions of all metrics, including the standard ones, were placed in the appendix, not just the LSM.

(2)The LSM evaluates the degree to which concept-level explanations satisfy predefined logical rules. If a rule holds under the current concept prediction, the corresponding potential function for that rule has a value of 1, otherwise it is 0. We compute the weighted ratio of the potential functions with a value of 1 to all potential functions in the factor graph.The LSM is defined as the mean of the weighted ratios across all the test samples.

(3)We compute the LSMs of other methods based solely on their generated concept activations. These activations are assigned to our factor graph, which is then used to compute the LSM. Notably, other methods do not need to construct a separate factor graph for this process. To ensure experimental fairness, all methods utilize the same factor graph during the LSM computation phase.

2.

Thank you for your further comment.

(1)E-ACC and P-ACC denote the predictive accuracy of concepts and the predictive accuracy of tasks, respectively. Therefore, we can simply understand that E-ACC is the accuracy of explanation and P-ACC is the accuracy of final prediction. In datasets, each sample contains task prediction labels and concept prediction labels. Task prediction labels are stored as indexes of categories and concept prediction labels are stored as binary vectors. Each element of the vector indicates whether each concept is active or not (whether the concept exists in the sample). Therefore, we calculate the similarity of labels and predictions to compute the E-ACC and P-ACC. In fact, we have already described this in the paper (Line 1054-1062), but we would like to clarify the reviewers' concerns by providing more details.

(2)When there is no perturbation in the environment, concept predictive accuracy is positively correlated with task predictive accuracy. In CBMs, the input of the category predictor is concepts. Therefore, a high concept predictive accuracy enhances the higher the category predictive accuracy as well. However, in the adversarial perturbation setting, an attacker can tamper with several concepts that will not affect the final prediction, resulting in a significant reduction in concept accuracy while the prediction accuracy remains almost unchanged. Moreover, LSM is positively correlated with concept predictive accuracy because concept labels typically satisfy most critical logical rules.

(3)In fact, we made a lot of efforts in order to put the comparison results of all metrics in the main paper. However, due to space constraints in the paper, we had to put the results of the comparison between E-ACC and P-ACC (this contains 6 tables) in the appendix. We will readjust the layout of the camera-ready version so that we can put the results of the E-ACC and P-ACC comparisons into the main paper. But even so, we would like to emphasize that the comprehensibility of explanations naturally depends on the extent to which they satisfy human perception ( the logical rules of the real world) [4]. Thus, the results of the LSM comparisons are a straightforward demonstration of the advantages of AGAIN in enhancing the comprehensibility of explanations.

Weaknesses 1:

We fully understand your concern about manual annotations. However, this is widely considered as a limitation of datasets, rather than of CBMs [2]. CBM itself does not require an annotation process; therefore, AGAIN simply utilizes the well-established concept labels available in datasets. For concept annotation of datasets, many studies have already designed plug-ins to automatically annotate concepts specifically for datasets without concept labels [1-4]. These plug-ins use a multimodal-based approach to automatically extract concepts from samples and have achieved excellent performance. These plugins are sufficient to provide data support for CBMs training.

Weaknesses 2:

We apologize for any confusion Section 4.2 may cause. We have made an adjustment to Section 4.2 in the Rebuttal Revision (Line 230-246). The phrase 'reasons about a conditional probability' is a typographical error, and $G$ refers to a graph.

Weaknesses 3:

(1) The datasets selected in this paper reflect adversarial scenarios in practice. For example, CUB has been a standard dataset for evaluating the prediction accuracy and interpretability of CBMs under adversarial attacks [5-8].

(2) AGAIN is applicable to real-world situations. AGAIN was evaluated on real datasets capable of modeling real-world situations. Please allow us to give some existing examples to prove that these datasets are capable of modeling real situations, verifying the applicable of our model:

• Mazumder et al. [10] argue that CUB can effectively evaluate “the performance of deep models in real life situations”.

• Jiang et al. [11] considered the CUB dataset as a “fine-grained dataset in real-world applications”, and argued that evaluating on CUB "can contribute to advancement of the proposed approach in real-world scenarios".

• As presented in [9], "the within-class variance in the CUB dataset is large due to variations in lighting, background, and extreme pose variations (e.g., flying birds, swimming birds, and perched birds partially obscured by branches)". These characteristics provide a strong simulation of real-world situations.

(3) Real-world adversarial scenarios can be summarized into two types: 1) sensor disruption: an attacker can remotely disturb data sensors to perturb the data signals fed to the model (e.g., disrupting a patient's e-health metrics); 2) data tampering: an attacker can directly modify the data by adding a perturbation and re-feeding it back to the model (e.g., putting a disruptive texture on a road signage to misdirect the in-vehicle camera). The real datasets we use can fully cover the above two realistic adversarial scenarios. Specifically, the samples of the MIMIC -III EWS dataset which are injected with perturbations can simulate the sensor signals that are interfered by the attacker in Scenario 1. The samples of the CUB dataset injected with perturbation can simulate the disruptive texture imposed on the image by the attacker in Scenario 2.

Questions:

Following your comments, we revised the manuscript to provide more details about probability estimation. Please refer to Line 230-246 in the Rebuttal Revision of the paper.

Firstly, after each variable (concept and category) in $\mathcal{G}$ is assigned a value, boolean values are output from potential functions of all factors. These boolean values indicate whether the assignments of concepts and categories satisfy the logical rules represented by potential functions. Therefore, the weighted sum of all potential functions quantifies the extent to which concept assignments satisfy the logic rules in $\mathcal{G}$. Then, we seek to obtain the likelihood of the current concept assignments occurring, conditional on the known categories and logic rules. We quantify this likelihood by computing a conditional probability using the weighted sum of potential functions. We consider all possible concept assignments and compute the expectation of current concept assignments. This expected value is considered as the conditional probability, which is then used to detect whether concept activations are perturbed. For illustrative purposes, we provide an example. Suppose there are concepts $A$ and $B$. The current concept assignment is {1,0} denoting $A=1$ (active) and $B=0$ (inactive). We iterate through all four possible assignments: {1,0}, {0,1}, {1,1}, {0,0}. We compute the weighted sum of the potential functions for each of the four cases and compute the expectation of the potential function for {1,0}. This expectation is the conditional probability that concept assignment $\{1,0\}$ occurs conditionally on the known categories and logic rules.

References:

[1]Label-free concept bottleneck models. ICLR.

[2]Discover-then-name: Task-agnostic concept bottlenecks via automated concept discovery. ECCV.

[3]Incremental Residual Concept Bottleneck Models. CVPR.

[4]Semi-supervised Concept Bottleneck Models. ICML.

[5]Understanding and enhancing robustness of concept-based models. AAAI.

[6]Concept bottleneck models. ICML.

[7]Addressing leakage in concept bottleneck models. NeurIPS.

[8]Interactive concept bottleneck models. AAAI.

[9]The caltech-ucsd birds-200-2011 dataset.

[10] Few-shot lifelong learning. AAAI2021.

[11] Navigating Real-World Partial Label Learning: Unveiling Fine-Grained Images with Attributes. AAAI2024.

After going through the paper details with rebuttal information, I think this paper clearly does not reach the ICLR acceptance bar. Relying on pre-defined concepts and logics is the major concern. The approach is not novel or practically scalable. Another minor problem is that the paper's presentation still needs to go through some major revision before being considered publishable.

Weaknesses 1:

Thank you for your comments. We understand your concern about the reliability of our assumption. Actually, no. This assumption is realistic. Predictions are erroneous when they are affected by perturbations. We’d like to emphasize that most studies in the interpretability domain mainly focus on defending against perturbations that affect the explanation without changing the predictions [2]. This focus is due to the fact that, when the predictions are incorrect, the pursuit of explanations becomes less meaningful [3]. This is because ambiguity arises only when correct predictions are paired with incorrect explanations. Regarding your concern, we’d like to give a more detailed explanation of this situation: For concept bottleneck structures, attackers can generate perturbations that only interfere with concepts but not predictions. For example, [1] claims that “An attacker can easily disrupt concepts without changing the prediction label.” A simple way to achieve this is to optimize an objective function to maximize the difference in concept activation while ensuring that the final prediction remains unchanged. This objective function converges with only a few optimization steps. The attacker can then search the solution space for perturbations that meet the condition. As a result, attacks based on these perturbations are easy to implement and widespread.

Weaknesses 2:

External knowledge is typically expressed as logical rules. However, these rules are often discrete. Factor graphs can incorporate these discrete rules, enabling the detection and correction of wrong explanations. Known logic rule sets can only handle strictly deterministic reasoning (either established or unestablished). This limitation makes purely logic-based reasoning inaccurate, as real-world external knowledge is often uncertain or fuzzy. To accurately detect erroneous interpretations, it is essential to reason with uncertainty about this external knowledge. Factor graphs enable this by leveraging rule weight learning and conditional probability estimation, thereby enhancing detection accuracy. In contrast, while a collection of known logic rules can also detect erroneous explanations, its detection accuracy may be limited due to the absence of uncertainty reasoning. To validate this, we have included a set of ablation experiments in the Rebuttal Revision (Line 1119-1133), comparing the effectiveness of factor graphs with purely logic rule sets. The experimental results demonstrate that factor graphs significantly outperform methods that rely solely on logic rules.

Weaknesses 3:

The explanation of Figure 3 is insufficient. There is no reference to Figure 4. Thank you for pointing out these issues. We have revised the manuscript based on your suggestions. You may refer to Line 209 and Figure 3.

Questions 1:

Constructing a bipartite complete graph is a straightforward way. But the overhead of constructing a bipartite complete graph is the same as AGAIN. After constructing the bipartite complete graph, external knowledge still needs to be injected by setting weights based on logical rules. If logical rules are completely omitted and weights are learned solely from data samples, the graph fails to improve the comprehensibility of the explanation, as it lacks external knowledge necessary for detecting and correcting incorrect concepts. Therefore, to ensure the comprehensibility of the explanation under unknown perturbations, we must construct the factor graph using logical rules.

Questions 2:

Each factor is defined as a potential function that performs logical operations based on different rules, which can be categorized into coexistence and exclusion operations. We explained this point in the paper (Line 216-221) and apologize for any confusion caused by our initial explanation. Additional clarification has been included in the Rebuttal Revision (Line 213-215).

Questions 3:

Thank you for your suggestion. We have made additions in the Rebuttal Revision of the paper (Line 1097).

Questions 4:

Attributional training focuses on retraining the model with adversarial samples to maximize the similarity between the model's explanations and the explanatory labels. Therefore, attributional training can be considered a type of adversarial training. we have made adjustments in the Rebuttal Revision (Line 358).

[1] Understanding and enhancing robustness of concept-based models. AAAI, 2023.

[2] Adversarial attacks and defenses in explainable artificial intelligence: A survey. Information Fusion, 2024.

[3] Interpretation of neural networks is fragile. AAAI, 2019.

[4] Enhanced regularizers for attributional robustness. AAAI, 2021.

Dear reviewer XsjD,

We really appreciate your effort in reviewing our paper, your suggestions are highly valuable to us. As the rebuttal phase is nearing its end, we want to kindly follow up to ensure there is sufficient time for any further clarifications or concerns you might have. Please let us know if there is anything further we can do to assist or elaborate on.

Thank you once again for your time and effort in reviewing our paper.

Best regards,

The authors of submission 7228