Towards Multi-dimensional Explanation Alignment for Medical Classification

We appreciate the reviewers' recognition of our work and your constructive comments. Please find below our detailed responses to the queries you have raised. We hope you could consider increasing our score if our response addresses your concerns :)

W1: "captions and notations..."

Response: We have added notations to Figure 2, 3 and also enhanced the captions for Figures 2, 3, and 4 to improve readability. We have placed the modified table with the notation changes in the PDF. The corresponding original equations have the following modifications:

• On line 131: N -> M
• On line 133,135,136: k -> $d_c$
• On line 130: c = {$p_1, \cdots, p_k$} -> $\mathcal{C} =${$C_1, C_2, \dots, C_N$}
• On line 173: $\mathcal{C} =${$c_1, c_2, \dots, c_N$} -> $\mathcal{C} =${$C_1, C_2, \dots, C_N$}
• Throughout the text, replace all heatmap H with P.
• On line 184: $H_{p,k,i} = \frac{t_i^T V_{p,k}}{||t_i||\cdot ||V_{p,k}||},\quad p = 1, \ldots , P, \quad k = 1, \ldots , K$ -> $P_{h,w,i} = \frac{t_i^T V_{h,w}}{||t_i||\cdot ||V_{h,w}||},\quad h = 1, \ldots , H, \quad w = 1, \ldots , W$
• On line 189: $s_i = \frac{1}{H \cdot W} \sum_{h=1}^{H} \sum_{w=1}^{W} P_{h,w,i}$
• On line 195: $c =$ {$C_1, \ldots, C_M$} -> $c =${$c_1, \ldots, c_M$}
• On line 212: $D$ -> $X$ (Here, the dataset and the dimensional representation in the previous text conflict)
• On line 215: $f_c(x_{m}), \hat{c_m}, f(x_m) = \Theta_c(\Theta_b(x_m))$ for $i \in [M]$ -> $f(x_m) = \Theta_b(x_m), f_c(x_{m}),\hat{c_m} = \Theta_c(f(x_m))$ for $i \in [M]$
• More detail of $I_{o,j}$ and $I_{r,j}$

W2: "It is better to give pseudo codes of training and inferences..."

Response: The pseudocode has been supplemented, as seen in Figure 2 of the PDF.

W3: "At the bottom of Page 3..."

Response: It has been demonstrated that unidimensional interpretive aids such as saliency map [1,2], concept [3,4], and other information will contribute to medical decision making. To the best of our knowledge, our work is the first to propose a deeper aiding strategy for medical aids using multidimensional interpretable means. The advantage of this multidimensional aid is not only in the cumulative nature of the multiple facilitation strategies but also in the fact that when there is an error in one dimension of interpretability, the information from the other dimensions is able to correct that erroneous information, thus providing more accurately interpreted information.

In addition, we also combine neural symbolic and fuzzy logic to further the connection between the concepts, which makes the interpretation of concepts more auxiliary meaning.

W4: "In line 131, does $y_i$ the one-hot labels?"

Response: Yes, $y_i$ is the one-hot labes.

W5: "It is not clear why the authors use "average pooling to heatmaps"..."

Response:

Thank you for the valuable suggestions. Our approach is based on experimental results, which show that average pooling is comparable to or even better than other methods.

W6: "Under the line 183, the $V$ is three-dimensional, and why here it has two subscripts."

Response: Yes, it is a 3-dimensional tensor. Here, we use $V_{h,w}$ to represent $V_{h,w,\cdot}$, which is a vector whose length is $D$. The term $V_{h,w}$ is dot-multiplied with $t_i^T$ to obtain an intermediate result of size $\mathbb{R}^{D \times D}$. We then apply pooling to obtain the result for the heatmap at the position $(h, w)$.

W7: "In line 194, does the representation be ...."

Response: The meaning of $c$ is the set of all image concept labels, where $c=${$c_1, . . . , c_M$}. This notation is correct, with each $c_i \in${$0, 1$} $^N$.

W8: "In line 142, the logic symbols should be defined may be in supplementary."

Response: Thank you. We will define them in supplementary. For example:

• Negation ($\neg$): This symbol represents the inverse of a truth value. In fuzzy logic, the negation of a degree of truth $x$ is computed as $\neg x = 1 - x$. For example, if $x$ represents a 0.7 probability of a statement being true, then $\neg x$ would be 0.3, indicating a 0.3 probability of the statement being false.
• T-norm (Conjunction, $\land$): The t-norm, or triangular norm, is a continuous version of the AND operation. It takes two truth values $x$ and $y$ and combines them to produce a new truth value in the range $[0, 1]$. Common t-norms include the minimum function ($\min(x, y)$) and the product function ($x \cdot y$). In the example given, $c_{GO} \land \neg c_{LC}$ would represent the conjunction of "GO" being present and "LC" not being present.
• T-conorm (Disjunction, $\lor$): The t-conorm, or triangular conorm, is a continuous version of the OR operation. It also takes two truth values $x$ and $y$ and combines them, typically using functions like the maximum function ($\max(x, y)$) or the sum function ($x + y - xy$), ensuring the result is within $[0, 1]$.

W9: "Lack of training details..."

Response: To highlight the significant performance of our method, we did not use learning rate decay; the learning rate was kept constant at 5e-5 throughout the entire training process. For data augmentation, we only used the most basic operations: resize (256), center crop (224), and normalization.

Reference

[1] Van der Velden et al. "Explainable artificial intelligence (XAI) in deep learning-based medical image analysis." Medical Image Analysis 79 (2022): 102470.

[2] Patrício et al. "Explainable deep learning methods in medical image classification: A survey." ACM Computing Surveys 56.4 (2023): 1-41.

[3] Patrício et al. "Coherent concept-based explanations in medical image and its application to skin lesion diagnosis." CVPR 2023.

[4] Eminaga et al. "PlexusNet: A neural network architectural concept for medical image classification." Computers in biology and medicine 154 (2023): 106594.

We appreciate the reviewers' recognition of our work and your constructive comments. Please find below our detailed responses to the queries you have raised. We hope you could consider increasing our overall score if our response addresses your concerns :)

W1: "The caption of the picture is not detailed enough, as shown in figure 3."

Response: Thank you for your suggestions. We have provided additional details for each figure caption as follows:

• Figure 2: (a) module, output rich dimensional interpretable conceptual information for the specified disease through the multimodal model and convert the conceptual information into text vectors through the text embedding module; (b) module, access the image to the image embedder to get the image features, and then match with the conceptual textual information to get the relevant attention region. Then, we get the influence score of the relevant region information through pooling, and finally send it to the filter to sieve out the concept information with weak relevance to get the disease concept of image information.
• Figure 3: Overview of the Med-MICN Framework. The Med-MICN framework consists of four primary modules: (1) Feature Extraction Module: In the initial step, image features are extracted using a backbone network to obtain pixel-level features. (2) Concept Embedding Module: The extracted features are fed into the concept embedding module. This module outputs concept embeddings while passing through a category classification linkage layer to obtain predicted category information. (3) Concept Semantic Alignment: Concurrently, a Vision-Language Model (VLM) is used to annotate the image features, generating concept category annotations aligned with the predicted categories. (4) Neural Symbolic Layer: After obtaining the concept embeddings, they are input into the Neural Symbolic layer to derive conceptual rules. Finally, the concept embeddings obtained from module (2) are concatenated with the original image embeddings and fed into the final category prediction layer to produce the ultimate prediction results.
• Figure 4: A full comparison of our method with current saliency and CBM methods in terms of interpretable dimensions and effectiveness is shown in Fig. 4. CBM class methods lack interpretation of saliency map and rules dimensions. saliency map class methods lack interpretation of concept and rules dimensions. For this medical classification task, single dimensions such as saliency map methods do not accurately capture the features of disease images, and CBM-type methods for concept extraction of image information carry false predictions. And both of them lack explanatory information about the conceptual rules. Our method is able to predict interpretable information in three dimensions: saliency map with concept and rules, which provides more accurate multi-dimensional information, as reflected in the more accurate capture of disease features in image information, clearer concept output, and logical rule information generated based on the predicted concepts.

W2: "The author considers interpretability on the basis of concept embedding, neural-sybolic reasoning and saliency map. The author lacks the reason why it is necessary to analyze the interpretation from these aspects and whether there are other more dimensions."

Response:

concept: This approach focuses on understanding the internal representation of high-level concepts in a model. By analyzing the learned embeddings, researchers can gain insights into how the model associates different input features and concepts. It helps in understanding the semantic meaning of the model's hidden layers and can reveal if the model has learned meaningful abstractions.

neural-symbolic: This combines the power of neural networks with neural symbolic logic, aiming to make models more transparent by grounding their decisions in symbolic rules via concept. It allows for more explicit concept reasoning processes, which can be more interpretable than purely connectionist models.

saliency map: These highlight the parts of the input that are most influential in the model's decision. By visualizing these maps, one can see which features or pixels are driving the prediction, providing a more intuitive understanding of the model's behavior.

There are also indeed other dimensions:

(1) LIME (Local Interpretable Model-agnostic Explanations)

(2) SHAP (SHapley Additive exPlanations)

These are post-hoc explainability methods, and these techniques could be less scalable, less suitable for large models, and may struggle to align with specific tasks or samples.

Our framework, however, goes beyond these limitations. By integrating concept-based and attention map explanations, we offer a more intrinsic understanding of the model's decision-making process. Furthermore, the introduction of neural symbolic methods allows us to delve into clearer concept reasoning and decision-making, thus providing a third dimension of interpretability.

Specifically, we have successfully aligned the attention-based image features with concept-based semantics, ensuring a direct correspondence between the model's perception and the underlying meaning. Moreover, the neural-symbolic approach enables us to elucidate the interactions among concepts, which is crucial in understanding complex medical scenarios.

We appreciate the reviewers' recognition of our work and your constructive comments. Please find below our detailed responses to the queries you have raised. We hope you could consider increasing our overall score if our response addresses your concerns :)

W1,Q1: "Notations.."

Response: We have placed the modified table with the notation changes in the PDF. The corresponding original equations have the following modifications:

• On line 131: N -> M
• On line 133,135,136: k -> $d_c$
• On line 130: c = {$p_1, \cdots, p_k$} -> $\mathcal{C} =${$C_1, C_2, \dots, C_N$}
• On line 173: $\mathcal{C} =${$c_1, c_2, \dots, c_N$} -> $\mathcal{C} =${$C_1, C_2, \dots, C_N$}
• Throughout the text, replace all heatmap H with P.
• On line 184: $H_{p,k,i} = \frac{t_i^T V_{p,k}}{||t_i||\cdot ||V_{p,k}||},\quad p = 1, \ldots , P, \quad k = 1, \ldots , K$ -> $P_{h,w,i} = \frac{t_i^T V_{h,w}}{||t_i||\cdot ||V_{h,w}||},\quad h = 1, \ldots , H, \quad w = 1, \ldots , W$
• On line 189: $s_i = \frac{1}{H \cdot W} \sum_{h=1}^{H} \sum_{w=1}^{W} P_{h,w,i}$
• On line 195: $c =$ {$C_1, \ldots, C_M$} -> $c =${$c_1, \ldots, c_M$}
• On line 212: $D$ -> $X$ (Here, the dataset and the dimensional representation in the previous text conflict)
• On line 215: $f_c(x_{m}), \hat{c_m}, f(x_m) = \Theta_c(\Theta_b(x_m))$ for $i \in [M]$ -> $f(x_m) = \Theta_b(x_m), f_c(x_{m}),\hat{c_m} = \Theta_c(f(x_m))$ for $i \in [M]$
• More detail of $I_{o,j}$ and $I_{r,j}$

W2: "hyperparameters.."

Response: Thank you for your valuable feedback. We conducted experiments on reasonable parameter ranges for λ1 and λ2, as detailed in the table below. The table shows the accuracy of various backbones with different combinations of λ1 and λ2. Notably, the combination of λ1=0.1 and λ2=0.1 (used in the paper) performs well overall. When λ1 and λ2 are too small, accuracy generally decreases, indicating that λ1 and λ2 play a positive role in model optimization. Conversely, as λ1 and λ2 increase, different backbones exhibit varying performance. Overall, λ1 and λ2 values between 0.1 and 1 yield good model performance and show significant improvement compared to the baseline.

We conducted experiments to determine the appropriate threshold values. We sampled 100 samples and the results are as follows:

Q2: "all training images.."

Response: We annotated 20% of the training data for fine-tuning the model and labeled all images, including both training and testing images. This is because we need to use the automatically generated concept labels during inference.

Q3: "equation.."

Response: Thank you for pointing this out. The expression in our paper was somewhat unclear. We have revised it as follows: For each image $x_m$, after inputting it through the backbone, we obtain $f(x_m)$, represented as $f(x_m) = \theta_b(x_m)$, where $f(x_m) \in \mathbb{R}^{d}$. Next, we input the image feature $f(x_m)$ into the concept encoder $\theta_c$. It is important to note that $\theta_c$ returns two values: one is the predicted concept $\hat{c}_m$ and the other is the concept feature $f_c(x_m)$. The dimensions of these are $\hat{c}_m \in${$0, 1$}$^N$ $\in \mathbb{R}^{d_c}$ and $f_c(x_m) \in \mathbb{R}^{d_c}$, respectively. In the paper, $d$ and $d_c$ is set to 1000, and $N$ depends on the number of concepts.

Q4: "during training.."

Response: During training, all modules are trained together. As shown in Table 1, the result of using only the neural-symbolic layer (DCR) is very poor, indicating that joint training is necessary. It is also worth noting that while $\mathcal{L_{neural}}$ primarily affects the output of the neural-symbolic layer, $\mathcal{L_c}$ impacts the concept prediction results, and $\mathcal{L}_{task}$ affects the classification results. Due to the nature of backpropagation, these losses also optimize the concept encoder and backbone. As evident from the ablation studies in Table 2, removing any of these losses leads to a decline in overall model performance.

Q5: "Figure 4.."

Response: A full comparison of our method with current saliency and CBM methods in terms of interpretable dimensions and effectiveness is shown in Fig. 4. CBM class methods lack interpretation of saliency map and rules dimensions. saliency map class methods lack interpretation of concept and rules dimensions. For this medical classification task, single dimensions such as saliency map methods do not accurately capture the features of disease images, and CBM-type methods for concept extraction of image information carry false predictions. And both of them lack explanatory information about the conceptual rules. Our method is able to predict interpretable information in three dimensions, saliency map with concept and rules, which provides more accurate multi-dimensional information, as reflected in the more accurate capture of disease features in image information, clearer concept output, and logical rule information generated based on the predicted concepts.

We appreciate the reviewers' recognition of our work and your constructive comments. Please find below our detailed responses to the queries you have raised. We hope you could consider increasing our overall score if our response addresses your concerns :)

W1, W2: "evaluation of the interpretability"

Response: The questions you raised are very insightful. Although quantifying interpretability remains a challenging task with no well-established metrics, our extensive examples demonstrate that the interpretability information provided by our model is meaningful. This is supported by numerous examples presented in our paper, particularly in Figures 1 and 5-12. While aligning saliency maps with features has been a longstanding issue in interpretability, we have compared Med-MICN with baseline saliency maps. As illustrated in Figure 1 of the attached PDF, we used the most distinctive skin modality for clarity. The results show that our method better highlights pathological regions and achieves a significant improvement in alignment compared to the baseline.

W3: "For section 4.3, the description of Neural-Symbolic Layer is not easy to follow....".

Response: Concept Polarity focuses on determining the role of a concept as either positive or negative for a given prediction class. The output of the neural network $\Phi_j(\cdot)$ is a soft indicator value between 0 and 1, which signifies the degree to which the concept contributes positively (closer to 1) or negatively (closer to 0) to the prediction. For example, if the output for the concept is 0.8 for class $j$, it suggests that this concept has a strong positive influence on predicting class j.

Concept Relevance assesses how useful or significant a concept is within the context of the sample feature for a prediction class. The neural network $\Psi_j(\cdot)$also outputs a soft indicator value between 0 and 1, but this value represents the concept's relevance score, indicating how much it impacts the prediction. If the output for the concept is 0.2 for class $j$, it means that this concept has a relatively low relevance to predicting class j.

While both functions output scalar values between 0 and 1, Concept Polarity indicates the concept's positive or negative influence, and Concept Relevance measures the concept's overall importance or significance in the prediction. These two outputs are combined in Equation (2) to generate the logical reasoning rules for the concept.

Q1: "The authors used LLM (GPT-4V) in their proposed method. Would it be better..."

Response: Thanks to your suggestion, we have fully considered the generalization and accuracy of different diseases when using multimodal models for questioning. Here, we have added two medical multimodal models for comparison experiments (Due to word count limitations, the results from each model will be uniformly presented in the appendix of the paper). The first model is XrayGPT, The XrayGPT model is limited by the form of training data for different kinds of disease images, such as plain view lung CT map (dataset COVID) can not output accurate medical conceptual information. Moreover, for chest CT images, its prediction about different classes of viruses such as COVID output information is also affected and lacks generalization to different disease types.

Besides, we also use the better generalized Llava Med model for comparison. For the COVID dataset, it is capable of obtaining certain medical category information, however, the output disease load categories are not comprehensive compared to the methods used in this framework, and the comprehensiveness of the information.

Q2: "How do authors deal with different numbers of concept sets from the LLM? ...."

Response: It is important to note that each concept within the concept set is processed individually by the text encoder to obtain a corresponding text embedding, rather than inputting the entire concept set simultaneously. The LLM handles varying input sizes through different lengths of concepts, such as "Peripheral ground-glass opacities" and "Bilateral involvement." By obtaining the text embedding for each individual concept, we can derive a concept heatmap for each image and subsequently assign the appropriate concept labels.

Q3: "Did authors try with different λ1 and λ2?"

Response: Thank you for your valuable feedback. We conducted experiments within reasonable parameter ranges for λ1 and λ2, as detailed in the table below. The table shows the accuracy of various backbones with different combinations of λ1 and λ2. When λ1 is fixed, a larger λ2 is preferred, and similarly, when λ2 is fixed, a larger λ1 is preferred. This indicates that each loss function contributes to the model’s predictive accuracy. Additionally, we need to adjust these coefficients for different backbones to achieve optimal performance. For example, with VGG, setting λ1 = 1 and λ2 = 0.1 resulted in a 7.41% increase in accuracy.