A Unified Concept-Based System for Local, Global, and Misclassification Explanations

• The contextualization of the paper in terms of prior concept-based methods is missing important and relevant methods.[1-2-3]
• Overall the evaluation is unconvincing. While the method is framed as an automated concept-based method, it does not compare itself to any other automated concept-based method [1-2], so it is hard to judge the proposed method
• Minor weakness but I would advise dropping the notion of "ood" images in the methodology as here no samples are really ood in the true sense of the term (except arguably the segment but certainly not the image from other classes)

[1] Zhang et al. Invertible concept-based explanations for cnn models with non-negative concept activation vectors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11682–11690, 2021.

[2] Fel et al. Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.

[3] Fel et al. A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation. arXiv. 2023

The adoption of a model tailored for binary classification tasks in UCBS appears to be a point of concern. This configuration prevents the concept-based system from contrasting the target class against a specific non-target class, leading to a loss of information. This approach diverges from the operation of deep learning (DL) classification models, which are typically trained to contrast multiple classes concurrently using a softmax function.

A lot of details are missing. Is the binary DNN classifier a separately trained network? Or is it using the encoder of the original network (the neural network it is explaining) and adding an additional binary classification head?

The authors assert that UCBS obviates the necessity for user-defined concepts and the application of external scoring methods (i.e., it eliminates “the need for providing user-defined concepts and applying external scoring methods”). However, the binary DNN classifier employed in their approach could arguably be considered an external tool. Furthermore, the paper lacks clarity on the distinct advantages of UCBS in comparison to other established methods like TCAV.

This leads to a closely related significant concern: the absence of pertinent baseline methods. There exists a substantial body of work on concept-based explanations for deep neural networks (DNNs), yet none of these are included as baselines in the paper. Consequently, it becomes unfeasible to assess UCBS's merits relative to the state-of-the-art. It's worth noting that the baselines employed in the paper, such as SHAP and LIME, are general methods and not specific to concept-based approaches.

Additionally, the authors characterize UCBS as an unsupervised learning method, a claim that appears misleading. UCBS relies on a segmentation neural network that is trained in a supervised manner. The labels used for training these segmentation networks essentially become the inherited concepts for UCBS. Therefore, this claim about the unsupervised nature of UCBS does not seem to be accurate.

The local component of UCBS appears to closely resemble traditional local XAI methods, such as SHAP and LIME, by focusing on super-pixel relevance scores. However, it may be more insightful to explore the presence of global concepts within the local (prediction) context and their respective importance, akin to the methods introduced in CRP [1] and CRAFT [2]. Only the global UCBS component seems to be concept-based (by operating on the latent representations of the DNN).

In terms of the limitations of UCBS, a more comprehensive discussion regarding the number and relevance of hyper-parameters, particularly in the context of fine-tuning surrogate models, would provide a valuable addition. An ablation study investigating the influence of these hyper-parameters on the quantitative evaluation metrics would be of interest.

Concerning the quantitative evaluation, it's worth noting that the authors exclusively compare UCBS against local XAI methods. It would be beneficial to see a comparison against other concept-based XAI methods to provide a more comprehensive perspective. Additionally, the faithfulness evaluation curves are presented for only two methods and two samples in the appendix. Including the final curves used to calculate faithfulness metrics would enhance the overall value of the analysis."

[1] Achtibat, Reduan, et al. "From attribution maps to human-understandable explanations through Concept Relevance Propagation." Nature Machine Intelligence 5.9 (2023): 1006-1019.

[2] Fel, Thomas, et al. "Craft: Concept recursive activation factorization for explainability." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

• Concerns on implementation
  • It is mentioned that the output of the convolution, maxpool, and avgpool with zero pixel area is zero. However, the color of the padded area in the figure is black, which means that the value of the padded area will not be zero after input normalization. This brings out the concerns that the implementation differs from the intentions described on page 3.
• Underestimated performance of baselines
  • The deletion score of LIME and GradCAM of the paper (https://arxiv.org/pdf/1806.07421.pdf) is 0.12, which is much lower than the numbers denoted in this paper.
• Presentations
  • The datasets $\mathcal{D^c}$, $\mathcal{D^c_{S}}$, and $\mathcal{D^c_{out}}$ are defined as sets of images in the notation section, while they seem sets of $(x, y)$ tuples in Eq (2).
  • I guess from Figure 1a that $y = 0$ for $\mathcal{D^c_{out}}$ and $y = 1$ for otherwise, but it is not denoted in the paper.
  • Notation $c$ is sometimes omitted without mention.
  • I guess the $Net_S$ should be a set of functions with the same numbers as class, $Net_S=${$f^c_\theta$}.
  • The images in Figure 1 are too small.
• Absence of ablation study
  • I believe it is essential for this paper to show the ablation study, especially for concept learning. Specifically, we can replace the logit of surrogate model $f_\theta^{'c}$ as $IS(s_{i,j}) = f_\theta^{'c}(s_{i,j})$ with a pre-trained image classifier.

• The proposed approach is certainly not the first to explore the extraction of local and global explanations, please refer to [1, 2].
• What is the motivation for having the out-of-distribution dataset to train the surrogate model?
• The proposed approach only captures the concepts that can be described as a part of an image. Is it possible to capture concepts like lighting that are expressed in an entire image?
• I'm unsure of how to interpret global explanations. Can you please provide more discussion for the same?
• It is a bit unclear how the scores in Table 1 are computed, given the UCBS is fine-tuned on superpixel images; the scores like insertion, deletion, and faithfulness are expected to be high for UCBS, given they are on the manifold, while in case of LIME, SHAPE, GradCAM, and LRP the concepts are off-manifold resulting in lower score. Could you please justify this?

[1] Kori, A., Glocker, B. and Toni, F., 2022. GLANCE: Global to Local Architecture-Neutral Concept-based Explanations. arXiv preprint arXiv:2207.01917.

[2] Setzu, M., Guidotti, R., Monreale, A., Turini, F., Pedreschi, D. and Giannotti, F., 2021. Glocalx-from local to global explanations of black box ai models. Artificial Intelligence, 294, p.103457.