COMiX: Compositional explanations using prototypes

missing the discussion of several competitors.

Many thanks, we will add the appropriate citations for the presented work into the related work section. SOTA methods are actually not backbone-independent. ProtoPNet, for example, relies upon the standard convolutional architecture, so not compatible with B-Cos (see [4] for the details). I understand that for the same exact reason the more recent works such as [5] do not provide non-convolutional backbones. In summary, we do not compete with ProtoPNet-style models, but propose a different type of interpretable-AI model. The goal of the paper was not to propose a better ProtoPNet or address its shortcomings, but to produce high-fidelity interpretations using both the training data and inherent explanations. To address this concern, we would then like to spell out the differences between the proposed method and the ProtoPNet family. We would argue (and address it in the text more clearly) that actually ProtoPNet-type models offer post hoc explanation in the following sense. They optimise latent representations of the prototypes (in case of ProtoConcept [1], for example, they optimise the parameters of the prototypical balls. For these optimised latent representations, they either ‘project/push the testing example to the nearest training example‘ (ProtoPNet) or, in ProtoConcept, the authors visualise the training-data prototypes for the concepts contained within a ball. In contrast to this, the proposed method visualises the images which are directly forming a part of the decision-making process, thus providing fidelity of the explanations (see Steps 4 and 5 of Algorithm 1).

Notably, contrary to the reviewer’s claim, ProtoPool does not use exact training images as prototypes but uses projection of learnt prototypes into the training images just like ProtoPNet. Page 7 of Rymarczyk et al (2022) states: “Prototypes projection is a step in the training process that allows prototypes visualization. It replaces each abstract prototype learned by the model with the representation of the nearest training patch” It means that although they use certain other innovations, their works’ contribution is different from ours and they do not use exact training images for matching but instead use prototype projection, similarly to the other methods in the ProtoPNet family. “even without proper citation” We would like to kindly clarify, which particular work, in the opinion of the reviewer, has not been properly cited while being used in this work. As we discuss above, we do not propose a better ProtoPNet, but to provide inherent prototypical interpretation for the models inherently interpretable through attribution maps, such as B-cos. As the authors, we do not know of any such work, and we do not think it is similar, in its aims or goals, to ProtoPNet. But if you have any work in mind we would gladly cite this work if the reviewer comes up with it. Neither does this method constitute a combination of the methods. (1) As we state above, we respectfully disagree that ProtoPool implements a method similar to the proposed one, and neither of the suggested methods implement decision making through actual images. Furthermore, our method searches for the similarities in the features for the whole image and does not select image patches, as it is done in ProtoPNet. Implicitly, the patch-dependence makes ProtoPNet family methods bound to the convolutional models. The proposed method does not have such a restriction. (2) Although we use B-cos backbone, it may be used with the other methods, which provide attribution maps as a part of the computational pipeline. It may also be converted from the existing models, using, for example, the results from B-cosification paper [3]

Reported baselines are significantly too bad.

We would like to check where has this accuracy on ResNet34 been taken from and what is 2.7 p.p. more accuracy compared with ? We take our ResNet34 results, as stated in the paper, from Donnelly et al (2022), so it would be great if the reviewer could confirm the settings of their experiment.

Clarity of writing

Many thanks, we are going to update the notation to improve the clarity. A detailed explanation of Eq 2n3 are given in appendix 1. This is directly taken from Bohle et al. Table 4 is generated using the same protocol as the same experiment protocol as prior art (mentioned in the main text). We add Bcos to the table for comparison in the revised text.

In a more targeted way, let us look into the claim that ProtoPool may be the most similar method.

• Although the mechanism to obtain the activation maps is different from ProtoPNet, ProtoPool also produces prototype activation maps which reflect the similarity between the prototype and the input

• ProtoPool also uses the projection of the prototypes to the training image patches

• ProtoPool method is also focused on selection of learnable prototypes, and improves upon ProtoPNet

• ProtoPool does not produce per-feature explanation of decision making. Instead, it explains the whole decision through similarity with prototypes. We produce interpretations for every class-defining feature.

Therefore, we believe that ProtoPool is an entirely different method solving a different problem, addressing the architectural shortcomings of ProtoPNet such as per-class prototypes amongst others, which is the reason why we don’t present our work as an improvement upon ProtoPool. We will add the citation of ProtoPool into the review of interpretable methods. In terms of similar works within the ProtoPNet family, we would think Ma et al [5] is actually a closer match, as suggested and discussed with Reviewer HwsD.

Thank you for a prompt response and for such a thorough discussion about the concerns.

“This claim is not clear, as in this very paper, they use ProtoPNet with ViTs. In [4] they apply ProtoPNet to Vits and discuss reasons why it does not work as well, but it is still compatible. When looking at the Bcos-Vits (https://github.com/B-cos/B-cos-v2), it does seem like bcos-vits show a larger gap compared to the non-bcos versions too. So the entire motivation why this method is forced to use Bcos-ViTs and why it therefore compares with no competitors is not clear. Especially, why would ProtoPool or ProtoPNet not be compatible with a B-cos CNN?" Without proper citation refers to no citation for the works that are most similar, especially ProtoPool. I am also still not sure if I get the novelty compared to Protopool. As you cited, Protopool replaces prototypes with the representations of training patches. Thus, it exactly computes the similarity in its representation between a test image and a training image. Is the main difference from an interpretability point in patches as opposed to entire training images?”

We will try and clarify upon the difference between our work and ProtoPNet-style models. In short, we neither see our work similar to ProtoPNet beyond the similarity in the presentation of explanation, as both architecture and the problem is different. In summary, we believe that ProtoPNet family is just a different family of models, which is only related by the means of ptresentation of explanations through training data. The activation maps in ProtoPNet represent the similarity with a patch (i.e., it is a function of a patch and an input), while in our method we present the attribution for a particular feature (i.e., it is a function of an input and the feature id, not a similarity) so they are not like-for-like. The differences between the ProtoPNet models and the proposed model as per the ProtoPNet paper:

• ProtoPNet obtains the activation maps using the following procedure : This activation map preserves the spatial relation of the convolutional output, and can be upsampled to the size of the input image to produce a heat map that identifies which part of the input image is most similar to the learned prototype. We take advantage of self-interpretable architectures such as B-cos, which can produce attribution maps for any layer and any feature by-design, directly of a size of an input image. Hence, our method’s attribution maps are not related to the prototypes and only to the input, while the ProtoPNet’s activation maps are related to the similarity with learnable prototypes.

• There is a difference between the interpretation of the ProtoPNet’s activation maps and the COMiX’s attribution maps. ProtoPNet maps represent the highest activation for a particular prototype. COMiX’s maps represent the activation for a neural-network feature (and are thus only dependent upon the input data).

• ProtoPNet learns the set of per-patch prototypes (from ProtoPNet paper, Section 2.2: “we jointly optimize the convolutional layers’ parameters wconv and the prototypes P“). We do not do this, we attribute the decisions to the whole set of training data. We are motivated by the fact that we need to connect the decision with the training data.

• ProtoPNet does not connect the decisions directly with the training data, it connects it with the synthetic prototypes, learnt from the training data. It is a substantial difference: the prototypes in ProtoPNet are optimised with the training data matches found post hoc, while we directly express the decision through the training data. ProtoPNet uses the following technique for visualisation: “To be able to visualize the prototypes as training image patches, we project (“push”) each prototype pj onto the nearest latent training patch from the same class as that of pj . In this way, we can conceptually equate each prototype with a training image patch” It means that these nearest latent training patches in ProtoPNet only approximate the prototypes. Thanks to the by-design explanation provided by B-cos family models, we avoid such approximation.

First of all, we would like to thank the reviewer for their constructive feedback.

novelty isn't significantly

We would think that there are even more related methods from the ProtoPNet family. The more recent and more relevant paper, as also suggested by Reviewer HwsD, would be Ma et al (2024) (ProtoConcept), which performs visualisations for multiple image features in a prototypical setting. However, even this paper is not implementing the interpretation the same way. Their idea is to perform the explanations through multiple image patches. In the centre of our idea is the fact that inherently interpretable models, such as B-cos, can explain the decisions in different ways by every feature. In this way, our method bridges the gap between the concept bottleneck and the prototypical models.

We would then like to spell out the differences between the proposed method and the ProtoPNet family. We would argue (and address it in the text more clearly) that actually ProtoPNet-type models offer post hoc explanation in the following sense. They optimise latent representations of the prototypes (in case of ProtoConcept [1], for example, they optimise the parameters of the prototypical balls. For these optimised latent representations, they either ‘project/push the training example to the nearest training example‘ (ProtoPNet) or, in ProtoConcept, the authors visualise the training-data prototypes for the concepts contained within a ball. In contrast to this, the proposed method visualises the images which are directly forming a part of the decision-making process, thus providing fidelity of the explanations (Steps 4 and 5 of Algorithm 1). Furthermore, ProtoPNet-type models rely on selection of patches. It creates inherent bias to spatial proximity, and makes the model implicitly dependent upon the convolutional architecture as opposed to the alternatives such as transformers (see [2] for details why the ProtoPNet-type models cannot be used without modifications with the transformer architecture). The purpose of the paper has not been to provide an improved ProtoPNet architecture but to provide inherent interpretations in the style of ProtopNet explanation (this looks like that), without any post hoc analysis, through prototypes and using intrinsically interpretable models producing saliency maps, such as B-cos. We will update the text accordingly to provide better clarity upon this point, as it seems to be the common point between the reviewers.

Motivation/need for sufficiency is unclear

The motivation for sufficiency is highlighted by our goal to provide high-fidelity explanations and demonstrate that we achieve the goal of meeting the well-established desiderata for interpretable ML. To address it, in the next revision we will fix Line 96 to make it synchronised with the formalisation of sufficiency in Line 308. We believe it is standard for the interpretable ML literature (see, e.g. [3] which is defining sufficiency metrics for quantifying faithfulness. We will add further references highlighting the role of sufficiency in the related work.

Notation seems convoluted

The $s^i_{L}(x;\theta)$ are the per-feature components of $W_{1->L}(x;\theta)$, which we introduce to make Equation 6 more readable. We will refine the notation accordingly to improve the presentation.

Not entirely clear on the attribution method. Is it which feature a pixel contributes most to?

Many thanks for this, you are absolutely right on this. We will update the notation accordingly. The attribution is obtained by a truthful explanation of the encoding of the features using B-Cos models to provide the per-pixel contributions by-design (see Section 3.1 and Appendix A for more detail). We present the per-pixel attribution maps for a particular feature. See the answer to the question below on how exactly that map is produced.

We thank the reviewer for the comments and constructive feedback. Below we present the clarifications on the weakness and answer the questions.

W1: The computation of mutual information only involves precomputed features and the respective labels and not the explanation. This is not a computationally intensive process given the precomputed features and can be done offline, before the inference sample is processed. We add more details on the computation of mutual information in the main text.

As shown in figure 2, we divide our method into train and inference. At train time, we train the encoder, compute the train features and then the CDF features. At inference, computation of finding prototype examples takes KNN is a smaller feature space. This space is as small as one feature, which reduces to an argmax over the difference of N pairs of 2 numbers each. We add this to the description. We also use the label from the classifier as the pseudo label (added in the appendix). In fact any method that approximates the prediction can serve as a plug in for the getting pseudo label. We add another ablation to the appendix to show that the proposed method is robust to the performance of pseudo label prediction mechanism (Figure 12). Please note that using pseudo labels helps create an explanation such that ‘what makes a cat a cat is different from what makes a dog a dog’.

W2: First of all, we would like to spell out the differences between the proposed method and the ProtoPNet family. We would argue (and address it in the text more clearly) that actually ProtoPNet-type models offer post hoc explanation in the following sense. They optimise latent representations of the prototype (in case of ProtoConcept [2], for example, they optimise the parameters of the `prototypical balls’[2]. For these optimised latent representations, they either ‘project/push the training example to the nearest training example‘ (ProtoPNet) or, in ProtoConcept, the authors visualise the training-data prototypes for the concepts contained within a ball. In contrast to this, the proposed method visualises the images which are directly forming a part of the decision-making process, thus providing fidelity of the explanations (see Steps 4 and 5 of Algorithm 1). Our method, unlike the suggested baselines, takes decisions based on the explanation (not the other way round). Hence we get a reduced performance but at an advantage of high fidelity <refer table and numbers>. With methods such as B-cosification[5] we can use off the shelf models and then use them for inference in the proposed style, giving high fidelity explanations.

[5] Arya et al (2024) B-cosification: Transforming Deep Neural Networks to be Inherently Interpretable

W3: We use a B-cos network as the backbone and propose an inference method so that the explanation is part of the inference mechanism, i.e. the explanation is not post hoc. Instead of explaining the decision that is taken post-hoc through prototypes as it is done in ProtoPNet-type models, the decision making mechanism happens through a ‘this looks like that’-human understandable bottleneck. The trade-off between the inherent prototypical explanations on one hand, and the accuracy on the other hand is therefore the reason behind the inferior performance. On the end-to-end model, we see there might be two different interpretations to this: (1) training end-to-end COMiX model or (2) the B-cos backbone without pretraining-and-finetuning, but just trained from random weights. Could the reviewer clarify which one they mean? On interpretation (1), we would argue that the there are two class of methods to consider.

1. The attribution based explanation like B-cos that gives one global saliency map for a decision. various prior art has shown that this is a suboptimal explanation
2. The ‘this looks like that’ family of methods are post hoc in the sense they employ different mechanism for decision and explanation. Our method brings the best feature of both these explanation but we think of the the performance of the original B-cos model and the KNN on its feature space as the upper limit of the proposed method. On interpretation (2), we are now checking the performance of training from just random weights.

W4: The original intention was not to show the performance against the competitor, but to show the notional PQ value compared to the standard network. The comparison of sparsity in table is used to further motivate the use of B-cos encoder. Figure 6 shows the much higher sparsity of the proposed method. The performance of the method under as short as 12 features is reported. Also the method is sparse with respect to the number of train samples it uses for decision making (shown with the ablation with different K values). Also in the appendix we show Figure 8 which shows the performance of the method with features considered one at a time.

• Regarding the response to W1, I have already understood that mutual information computation is performed only in the precomputation phase, which helps to maintain overall computational feasibility. However, the authors' response could have been clearer in addressing my original concern about the mutual information computation process, which remains a key aspect in the proposed method.

• For the responses to W2 and W4, the authors stated that "...the proposed method visualizes the images which are directly forming a part of the decision-making process, thus providing fidelity of the explanations..." However, I find this argument unconvincing. The proposed method is aligned with the fundamental concept of prototypical learning—decisions are made based on the similarity between an arbitrary input and the learned prototypes. Thus, the lack of comparison with other prototype-based methods remains a significant concern. This omission is particularly notable as the authors themselves described the proposed method as a combination of object-level attention and prototypes, as presented in Table 3.

• For the responses to W3 and W5, while the authors emphasized the advantages of using the B-cos network for generating salience maps, prior research [1, 2] has highlighted limitations in the effectiveness of salience maps as meaningful explanations, especially in real-world applications such as medical domains. These limitations should be acknowledged and addressed.

• Thank you for providing clarifications on Q1 and Q2. I hope these points will be more clearly reflected in the revised manuscript. Despite the clarifications and responses, I still have several concerns after reviewing the responses and comments from other reviewers. Given the current state of the paper, I will maintain my original rating.

• References

• [1] Border, Samuel P., and Pinaki Sarder. "From what to why: The growing need for a focus shift toward explainability of AI in digital pathology." Frontiers in Physiology 12 (2022): 821217.

• [2] Venkatesh, Kesavan, et al. "Gradient-Based Saliency Maps Are Not Trustworthy Visual Explanations of Automated AI Musculoskeletal Diagnoses." Journal of Imaging Informatics in Medicine (2024): 1-10.

We thank the reviewer for the comments and constructive feedback. Below we present the clarifications on the weakness and answer the questions.

Originality: We would suggest that more recent and more relevant paper, as also suggested by Reviewer HwsD, would be Ma et al (2024) (ProtoConcept), which performs visualisations for multiple image features in a prototypical setting. However, even this paper is not solving the problem the same way. Their idea is to perform the explanations through multiple image patches. In the centre of our idea is the fact that inherently interpretable models, such as B-cos, can explain the decisions in different ways by every feature. In this way, our method bridges the gap between the concept bottleneck and the prototypical models, while, in contrast to ProtoConcept, avoiding altogether to explain it post hoc and instead providing interpretation at the same time as the decision. Below, we explain the last point in more detail, spelling out the differences between the proposed method and the ProtoPNet family.

We would argue (and we will address it in the text more clearly) that actually ProtoPNet-type models offer post hoc explanation in the following sense. They optimise latent representations of the prototypes (in case of ProtoConcept [1], for example, they optimise the parameters of the prototypical balls. For these optimised latent representations, they either ‘project/push the training example to the nearest training example‘ (ProtoPNet) or, in ProtoConcept, the authors visualise the training-data prototypes for the concepts contained within a ball. In contrast to this, the proposed method visualises the images which are directly forming a part of the decision-making process, thus providing fidelity of the explanations (see Steps 4 and 5 of Algorithm 1). We will update the text accordingly to provide better clarity upon this point, as it seems to be the common point between the reviewers.

[1] Ma, Chiyu, et al. "This looks like those: Illuminating prototypical concepts using multiple visualizations." Advances in Neural Information Processing Systems 36 (2024).

Quality: First of all, we would question why a model not being trainable end-to-end is a disadvantage per se. The selection of features and prototypes is a common problem between the concept- and prototype-based methods as the explanation over the whole feature set (e.g.., K nearest neighbours and 1024 features) is not viable. Nauta et al (2023), for example, dedicate Section 6.2 to describing the process of selection for the explanation size in the prototypical model setting. And to reiterate the point above, the principal difference between our setting and the ProtoPNet family of the models is that they find the training examples post hoc while we present our decision making through a set of training examples. This necessitates using only a handful of features. This, in a way, is a chicken-and-egg problem: we need to select class-specific values, but to do this, we need to know the class label. For that purpose, we need pseudo-labels to select the features.

Quality: We use a B-cos network as the backbone and propose an inference method so that the explanation is part of the inference mechanism, i.e. the explanation is not post hoc. Instead of explaining the decision that is taken post-hoc through prototypes as it is done in ProtoPNet-type models, the decision making mechanism happens through a ‘this looks like that’-human understandable bottleneck. The trade-off between the inherent prototypical explanations on one hand, and the accuracy on the other hand is therefore the reason behind the inferior performance.

Clarity: Many thanks for helping us improve the presentation of the paper. We will update the notation accordingly.

Significance: We think this stems from the suggestion that the method is, in fact, a variation of ProtoPNet. We do not think so: first, as we discussed above, ProtoPNet is not even the closest method out of the ProtoPNet family. Second, the principal contribution is the prototypical method which explicitly derives the decision from the training data, as opposed to post hoc assimilation of the similar training data to provide the explanation for the decisions.

Question: Many thanks for this comment, at the moment we are updating the notation to provide better clarity.