LucidPPN: Unambiguous Prototypical Parts Network for User-centric Interpretable Computer Vision

W1. In previous work, Huang et al. (2023) have discussed the inconsistency of traditional ProtoPNets. Does this issue exist within the proposed method? Please provide qualitative or quantitative evaluations.

To answer this question we have calculated consistency and stability of our method and compared it in Supplementary Table 11. One can observe that LucidPPN achieves a consistency score comparable to the method proposed by Huang et al. (2023) while outperforming other prototypical-parts-based methods. Regarding stability, LucidPPN demonstrates results on par with other methods. This improvement is likely due to the correspondence of prototypical parts to semantic parts of the classified objects.

W2. Please supplement the missing results for baseline methods on datasets like DOGS and FLOWERS in Table 1, as adapting to these datasets, which were not covered in the original papers, seems quite straightforward.

Thank you for your comment. We have added results for baselines on additional datasets, except for the ProtoTree. This exception is due to the model’s tendency to exhibit instability during training on these datasets. Upon reviewing the issues section of the ProtoTree GitHub repository https://github.com/M-Nauta/ProtoTree/issues, we noticed that others have faced similar challenges in applying this model to different datasets.

W3. This paper only implement the proposed method on several CNNs. However, vision Transformers are introduced to the realm of CV for several years, and have also been implemented as the backbone of ProtoPNets Xue et al. (2022). Please provide additional experimental results using ViT or even CLIP as the backbone.

Thank you for your comment. Unfortunately, we are unable to run LucidPPN with a ViT backbone for the following reasons:

• Incompatibility with PIPNet-Based Prototypical Part Definition: ProtoPFormer Xue et al. (2022) is built on the ProtoPNet-based definition of prototypical parts Chen et al. (2019), whereas our method relies on the PIPNet-based definition Nauta et al. (2023b). Currently, there is no adaptation of the PIPNet-based definition for the ViT backbone. This limitation is why we opted for the ConvNeXt backbone, which has demonstrated comparable performance to ViTs.
• Challenges with Self-Attention: Adapting a ViT backbone to the PIPNet-based definition is not straightforward due to the nature of self-attention. Unlike convolutions, self-attention lacks the properties of locality and a direct correspondence between the input and feature map Chen et al. (2019). This discrepancy makes it difficult to visualize prototypical parts faithfully.
• Orthogonal Research Direction: Adapting the ViT backbone to prototypical parts represents a separate research direction. Both ProtoPFormer and recent works in this area Ma et al. (2024a), which were unavailable at the time of submission, highlight that integrating a ViT backbone with prototypical parts is a non-trivial task requiring substantial architectural/training changes. For these reasons, we chose to use the ConvNeXt backbone in our work.

W4. In XAI, introducing human understandable semantics as evidence for prediction has been explored by concept bottleneck models (CBMs) Koh et al. (2020). What is the relationship between the proposed method and CBMs. Can concepts be introduced into the realm of ProtoPNet for higher interpretability?

Thank you for pointing out the connection between CBMs and our work. I’d like to clarify that both concept bottlenecks and prototypical parts can be considered concept-based models Bontempelli et al. (2022). However, there is a key distinction: concept bottlenecks use predefined intermediate classes (named concepts) that are directly associated with the image. While, prototypical parts, aim to identify relevant classification concepts during model training without any additional labels. A potential future research direction could involve combining concept bottlenecks with prototypical parts.

w1. What is the computational cost of inference and training? Please provide a comparison with baseline methods, including metrics such as training time, FLOPs, and memory usage.

In Supplementary Table 12 we provide information about training time, GFLOPs needed, and average memory usage during training for LucidPPN, PIPNet, ProtoPool, and ProtoPNet. One can observe that LucidPPN is faster and uses less memory than PIPNet. However, ProtoPNet and ProtoPool require much less memory to train while having longer training times.

Q1. The manuscript focuses on interpretability through the lenses of color, shape, and texture. However, other low-level features such as edges, contrast, and spatial frequency are also relevant. Have alternative low-level features also been considered in the analysis?

This work represents the first step toward disentangling low-level features. As mentioned in the Limitations Section, we plan to explore low-level features in more detail in future work. For now, we focus on extracting color information and correlating prototypical parts with semantic object parts. These contributions already make the work comprehensive. Introducing additional low-level feature extraction and integration at this stage would complicate the model further and make it more difficult to communicate.

Q2. The datasets utilized in the experiments are relatively small in size. How will the proposed method perform on larger datasets, such as ImageNet? Some insights into performance scalability would be beneficial.

Most benchmarking of prototypical part-based methods has been conducted on fine-grained datasets such as CUB and Stanford Cars Chen et al. (2019); Nauta et al. (2021; 2023b); Rymarczyk et al. (2021; 2022; 2023); Wang et al. (2024). Scaling these architectures to ImageNet-sized datasets is an important but orthogonal research direction that remains unsolved. However, to assess whether LucidPPN generalizes to broader classification tasks (beyond fine-grained datasets), we added results on PartImageNet He et al. (2022) in the Supplementary Materials. For this dataset, LucidPPN achieves an accuracy of 84.1%, outperforming PIPNet, which achieves 82.8%.

Q3. The manuscript primarily presents visualization results for the prototypical parts identified by the proposed method. How do these results compare with other prototypical parts-based models? A comparative analysis would enhance the understanding of the method’s effectiveness.

In Supplementary Figures 15-19, we added explanations from various prototypical-part-based methods. Additionally, our user studies demonstrate the effectiveness of our explanations from the user’s perspective.

Q4. In global feature visualizations, such as Figure 14, the manuscript illustrates the ability of the proposed method to detect shape and color. How does this compare with traditional edge detection operators (e.g., Sobel) for shape extraction and color feature extraction methods (e.g., color histogram)? Additionally, how does it compare with the direct visualizations of shallow layer attention to texture and color using techniques like Grad-CAM?

Edge detectors and color histograms are not trainable and do not represent high-level features, unlike prototypical parts. Even our low-level color prototypical representation captures a higher-level concept, such as a ”red tail,” which is then further decomposed into its components—color (red) and shape/texture (tail). Regarding GradCAM, we want to highlight that post-hoc methods can often be unreliable, as demonstrated in multiple studies Adebayo et al. (2018); Kim et al. (2022); Rudin (2019); Tomsett et al. (2020). This underscores the need for developing inherently interpretable models Rudin (2019); Rudin et al. (2022) such as our LucidPPN.

W1. While analyzing ”color,” ”shape,” and ”texture” offers a valuable perspective, these features have been extensively studied in the field of visual perception. Given that the shallow layers of deep networks are capable of extracting low-level features, the necessity for additional processing and analysis from prototypical parts raises concerns on the novelty and contribution of this work.

It is true that shallow layers of neural networks are capable of extracting low-level features. The goal of LucidPPN, however, is to make this processing more transparent to the user, which aligns with the broader objective of inherently interpretable models Chen et al. (2019); Rudin (2019). Thanks to LucidPPN we can analyze which colors were important for classification. Conventional neural networks often entangle visual features in ways that make it difficult to disentangle and present them in an understandable format for users. This is why we believe our work offers a novel contribution to the field of interpretable AI, particularly in the context of prototypical parts.

W2. The improvements demonstrated by the proposed method appear to be limited because its performance on some instances is lower than that of the compared methods

We respond to this comment in shared remarks in paragraphs The improvements demonstrated by the proposed method appear to be limited because its performance on some instances is lower than that of the compared methods. and There was no noticeable advantage in accuracy. Why?

W3. The organization of the experimental section appears somewhat unbalanced. While the results and visualizations presented are commendable, an excessive amount of content is relegated to the appendix, which may hinder the reader’s ability to grasp key insights and maintain a coherent narrative.

We agree and in the revised version of the manuscript, we have reorganized the experimental section. However, due to space constraints and to adhere to the ICLR template, some content has been moved to the appendix.

Q1. My main concern is that I did not see prototype projections in this work. Without prototype projections, how could you conclusively visualize prototypes using training images? The closest training images to a prototype could still be far away from the prototype in the latent space.

Our work builds on PIPNet’s definition of prototypical parts, which is why it lacks projection, which can lead to less faithful visualizations. Despite this drawback, PIPNet-based architectures have been successfully applied in various works De Santi et al. (2024a;b); Nauta et al. (2023a), and further developed, e.g. Wang et al. (2024) improving the interpretability.

Moreover, LucidPPN introduces a key difference in the definition of prototypical parts compared to PIPNet. While PIPNet employs Softmax across channels in the latent feature map, LucidPPN uses the sigmoid activation function. The sigmoid function allows each channel’s activation to be learned independently, not influenced by the relative activations of other channels. While, Softmax normalization can distort activations by emphasizing values that are only relatively high compared to others, even if they are low in absolute terms.

To build an intuition for this statement, let us consider $i$-th pixel of a feature map with activation values $z_i = [−2, 5, −0.2, −0.1]$, and $j$-th pixel with $z_j = [10, 300, 30, 10]$, the Softmax output $\theta$ for both would be $\theta_i = \theta_j = [0, 1, 0, 0]$. This implies that PIPNet would treat both pixels as equally important, despite the activations differing by a factor of 60. In contrast, with sigmoid activation $\sigma$ used in LucidPPN, the outputs would be $\sigma_i = [0.1192, 0.9933, 0.4502, 0.4750]$ and $\sigma_j = [1.0000, 1.0000, 1.0000, 1.0000]$, preserving the distinction in activation magnitudes. As a result, one can easily verify if the image patches selected for visualization are faithful because such patches should have a resemblance score close to 1.

Q2. During training, are the segmentation masks from PDiscoNet aligned with the ShapeTexNet feature maps or the aggregated feature maps?

Loss $L_D$ is applied only to the ShapeTexNet feature maps as we directly align them with masks from PDiscoNet. Indirectly, it also causes alignment of masks with the aggregated feature maps which are computed from the ShapeTexNet feature maps. To the Supplementary Materials (Figure 14), we added an image illustrating this process more concisely.

Q3. I am also not clear as to why binary cross entropy is used instead of multi-class cross entropy for training?

The intuition behind BCE usage is rooted from multilabel classification. To some degree ShapeTexNet operates in a multilabel setting from the prototypical parts perspective as they may match multiple classes. Hence, to enable multiple classes having high similarity to the same prototypical parts, we use sigmoid instead of softmax when computing the feature maps. This necessitates a shift from Cross-Entropy (CE) to Binary Cross-Entropy (BCE) because CE would then solely maximize the activation of the correct class while ignoring crucial signals from negative classes. Another reason behind our choice is to make it easier to verify the faithfulness of visualizations (see the answer about prototype projection).

Thank you for your response!

I see why you chose sigmoid (instead of softmax) to be applied to the latent feature maps.

"As a result, one can easily verify if the image patches selected for visualization are faithful because such patches should have a resemblance score close to 1."

In order to establish that your prototype visualizations are faithful, did you verify that every image patch selected for the visualization of a prototype has a self-resemblance score close to 1 with the prototype itself? Is it true? What would you do if you found a prototype whose visualized patch did not have a self-resemblance score close to 1?

Thank you for your clarification. I have raised my score to 6: marginally above the acceptance threshold.

To further improve the paper, it would be interesting to see what happens if you remove all prototypes whose self-resemblance score is not close to 1.

Also, it would be helpful to add a discussion on why you chose sigmoid (instead of softmax) and binary cross entropy (instead of multi-class cross entropy) for your method.

1. The Section 3 has a lot of paragraphs but lacks subheadings, making it difficult to follow the logical flow of the different parts.

We agree that Section 3 was dense. Therefore, we have revised it by introducing subsections that clarify the methodology of LucidPPN more clearly.

2. There was no noticeable advantage in accuracy. Why?

We answer this question in shared remarks in paragraphs There was no noticeable advantage in accuracy. Why? and The improvements demonstrated by the proposed method appear to be limited because its performance on some instances is lower than that of the compared methods.

References

J. Adebayo et al. Sanity checks for saliency maps. NeurIPS 2018.

A. Bontempelli et al. Concept-level debugging of part-prototype networks. arXiv 2022.

C. Chen et al. This looks like that: deep learning for interpretable image recognition. NeurIPS 2019.

L. A. De Santi et al. Patch-based intuitive multimodal prototypes network (pimpnet) for alzheimer’s disease classification. arXiv 2024a.

L. A. De Santi et al. Pipnet3d: Interpretable detection of alzheimer in mri scans. arXiv 2024b.

J. He et al. Partimagenet: A large, high-quality dataset of parts. ECCV 2022.

Q. Huang et al. Evaluation and improvement of interpretability for self-explainable part-prototype networks. ICCV 2023.

S. SY Kim et al. Hive: Evaluating the human interpretability of visual explanations. ECCV 2022.

P. W. Koh et al. Concept bottleneck models. ICML 2020.

C. Ma et al. Interpretable image classification with adaptive prototype-based vision transformers. arXiv 2024a.

C. Ma et al. This looks like those: Illuminating prototypical concepts using multiple visualizations. NeurIPS 2024b.

A. Nagrani et al. Attention bottlenecks for multimodal fusion. NeurIPS 2021.

M. Nauta et al. Neural prototype trees for interpretable fine-grained image recognition. CVPR 2021.

M. Nauta et al. Interpreting and correcting medical image classification with pip-net. ECAI 2023a.

M. Nauta et al. Pip-net: Patch-based intuitive prototypes for interpretable image classification. CVPR 2023b.

C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence 2019.

C. Rudin et al. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistic Surveys 2022.

D. Rymarczyk et al. Protopshare: Prototypical parts sharing for similarity discovery in interpretable image classification. ACM SIGKDD 2021.

D. Rymarczyk et al. Interpretable image classification with differentiable prototypes assignment. ECCV 2022.

D. Rymarczyk et al. Icicle: Interpretable class incremental continual learning. ICCV 2023

R. Tomsett et al. Sanity checks for saliency metrics. AAAI 2020.

BS Wang et al. Mcpnet: An interpretable classifier via multi-level concept prototypes. CVPR 2024.

M. Xue et al. Protopformer: Concentrating on prototypical parts in vision transformers for interpretable image recognition. arXiv 2022.