ProtoNMF: Turning a Black Box into a Prototype Based Interpretable Model via Non-negative Matrix Factorization

Thank you so much for liking our paper and we are highly encouraged by your positive and constructive feedback! It is our pleasure to address the primary concerns in the following sections in addition to the general rebuttal given above.

Adding an inference description Thank you for pointing this out. Your understanding is completely correct and we fully agree that it is helpful for readers to better understand our method. We have revised the section 3 of the manuscript to include this part.

How important the discriminativeness of prototypes is We agree that if the extracted prototypes are less discriminative, one may challenge their roles in the decision making. Thus we further propose a simple re-parameterization (detailed in general response above) to remove the residual prototype and increase the discriminative power of NMF prototypes, yet maintaining the interpretability of these prototypes.

What is missing in original NMF prototypes? A straightforward answer is the parameters in the residual prototype, because empirical evidence shows adding them can reach the original performance. A more intuitive understanding is: the features extracted by the pre-trained feature extractor are optimized to fit the pre-trained classifier, thus a precise reconstruction of the pre-trained classifier becomes important in maintaining the discriminativeness. However, in the general case, it's hard to obtain a precise reconstruction of a high-dimensional (e.g., 320 for coc-tiny, 512 for res34, 768 for vit-base) classification head using a basis consisting of small number of prototypes (e.g., 1,3,5,10), which is why the approximated head using only NMF prototypes does not perform well. Therefore, it's important that the missing part after the optimal reconstruction could be incorporated into the existing basis if one wants to express the classification head using exactly $p$ interpretable prototypes. The above analysis largely inspired the re-parameterization idea and we are very grateful to your in-depth and insightful comments!

We hope our rebuttal could address your concerns and sincerely hope that the reviewer could consider increasing the score.

We express our deepest gratitude for the reviewer’s time and constructive comments. It is our pleasure to address the primary concerns in the following sections in addition to the general rebuttal given above.

Interpretability advantage over black box+grad-cam Compared to Grad-CAM, although we could readily offer more detailed visualizations (e.g., multiple heatmaps for prototypical parts of each class instead of only a single heatmap for each class as in Grad-CAM), we argue that the visualization is neither our only advantage, nor the most important one. A much more important advantage is: Grad-CAM only tells what the network is looking at, but cannot tell what prototypical parts the focused area looks similar to, lacking the explanation on how these areas are actually used for decision [1]. In contrast, our method clearly tells what prototype the focused area looks similar to, how similar they are, and how important is that prototype in classifying an image to a certain class. Thus our method offers a much more transparent reasoning process by design. An example is given in section 4, last paragraph. Since Grad-CAM does not even have these capabilities, it's hard to show prototype based models are quantitatively more interpretable (section 1, paragraph 5 of our baseline [1] also claims this). Regarding the experiment on the potential usage for test-time intervention as a benefit: thanks for your suggestion! We add a case study in section A.4 of the appendix using re-parameterized NMF prototypes. This section demonstrates how human could intervene by setting the importance of the intervened prototype to zero using expert knowledge to correct the model's prediction. We show that assuming an expert can intervene one most important prototype once when correcting the prediction, the Top1 performance of coc-tiny can increase from $71.9\%$ to $82.7\%$ in ImageNet.

[1] Chen, Chaofan, et al. "This looks like that: deep learning for interpretable image recognition." Advances in neural information processing systems 32 (2019).

Writing and presentation Thank you for pointing this out! We have improved the method description in the revised manuscript. Regarding the diversity of prototypes: Note that we do not claim our prototypes to be more diverse than ProtopNet and we acknowledge that the perceptual diversity is hard to measure because the similarity of prototypes in the latent space doesn't necessarily match the human perceptual similarity [2]. Instead, we claim that they are more comprehensive (diverse yet complete, as explained in page 2, line 4) and offer quantitative results in Table 1. Seeking for pure diversity might not be that meaningful because even a set of random prototypes could be very diverse.

[2] Nauta, Meike, Ron Van Bree, and Christin Seifert. "Neural prototype trees for interpretable fine-grained image recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

The standard ResNet34 The result is already in the existing table. Our method only tries to decompose the classification head of a trained model for more transparent prototype based reasoning process, and thus has no influence on the performance.

Results in Table 3 We refer to the common response to all reviewers, where a simple re-parameterization could remove the not-so-interpretable residual prototype and increase the discriminative power of NMF prototypes, yet maintaining the interpretability of these prototypes.

We hope our rebuttal could address your concerns and we sincerely hope that the reviewer could consider increasing the score.

We express our deepest gratitude for the reviewer’s time and constructive comments. It is our pleasure to address the primary concerns in the following sections in addition to the general rebuttal given above.

Before addressing the reviewer's concern, we want to clarify two miss-understanding points based on the reviewer's summary.

Computation of the logit of class c is not based on the $E^c_{opt} B^c$, but is based on the $C^c_{opt}B^c$ [equation 4,5]. $C^c_{opt}$ are the coefficients optimized to make the NMF prototypes approximate the black box's classification head $V_c$ and the $E^c_{opt} \in \mathbb{R}^{nHW \times p}$ is used to indicate the interpretability of $B^c$ [section 3.1, 1st paragraph, last sentence].

The residual prototype is not added into each spatial position of each image via $H^c_{opt} R^c$. The $H^c_{opt} \in \mathbb{R}^{nHW\times 1}$ is used to indicate where and how strong the residual prototype is present in n images [section 3.2, paragraph below equation 6]. As pointed out by reviewer XoPR, the final classification head $V^c\in \mathbb{R}^{1\times D}$ decomposition can be expressed via rewriting the equation 5 as:

$

V^c = C_{opt}^cB^c+R^c

$

We apologize for causing the miss-understanding and have improved our writing in section 3 of the revised manuscript. We also added the Figure 8 to the appendix to compare the inference process of the black box, ProtopNet and our ProtoNMF.

How are latent prototypes which are not directly the feature of any image patch visualized It is visualized via indicating how important this prototype is to reconstruct the feature of each position of a set of real images. Given one prototype, such importance scores in all positions of one image together build one visualization. And combining such visualizations of all images makes human even easier to understand what exactly is important in these areas by observing common features in salient areas across all images.

Visualization of NMF prototypes The $E^c_{opt} \in \mathbb{R}^{nHW \times p}$ is used for visualization [section 3.1, $1^{st}$ paragraph, last sentence]. As a simple example, the first row in $E^c_{opt} \in \mathbb{R}^{nHW\times p}$ represents the $p$ coefficients of the first image patch (e.g., the upper left patch of an image) of the first image, and the first coefficient in this row indicates to which degree the feature of this patch can be approximated/reconstructed by the first prototype. So a large value in the $E^c_{opt}[1,1]$ means the first patch's feature is very similar to the first latent prototype. Combining the first $HW$ values of the first column tells us how similar each patch of the first image is to the first prototype. A visualization is shown in the dashed line's area of step 1 of Figure 1 of the manuscript, where the strength of the red color in the first bird image correspond to the first $HW$ red values in the first column of the $E^c$.

Visualization of the residual prototype. The visualization is similar. The only difference is: instead of looking at how similar the residual prototype is to the image patch's features, we look at how similar the residual prototype is to the image patch's "residual" feature. The "residual" feature indicates the feature that can not be modeled by the NMF prototypes [section 3.2, paragraph under equation 6]. For both types of prototypes, we apply a bilinear interpolation to upsample the resolution of the feature map to the resolution of the original image [section 3.2, last sentence]. At last but not least, we want to point out that a recently published paper by the ProtopNet's author also leverages the latent prototype and visualize them via a set of images instead of a single image patch [1].

[1] Chiyu Ma, Brandon Zhao, Chaofan Chen, Cynthia Rudin. This Looks Like Those: Illuminating Prototypical Concepts Using Multiple Visualizations, NeurIPS 2023.

Why this re-parameterization does not sacrifice the interpretability We focus the analysis on the difference between encoding matrix $E^c$ and $E^{c'}$ for interpretability visualization. Observe the equation for re-parameterization, we are actually shifting every axis of the coordinate system consisting of $p$ prototypes by the vector $\frac{R^c}{\sum_{i=1}^p a_i}$. Therefore, the change in the interpretability visualization is reflected by the change of the coordinates of all features in this coordinate system. For the ease of analysis, first assume each prototype is orthogonal to each other. Consider any prototype $b_i$, the strength of the $m^{th}$ feature patch $A^c_m$ from class c along the $k^{th}$ prototype $b_k$ is $A^c_m b_k^T$ before re-parameterization. After the re-parameterization, the projected strength becomes $A^c_m b_k^T + A^c_m (\frac{R^c}{\sum_{i=1}^p a_i})^T$. Therefore, the change of the visualization completely depends on the term $A^c_m (\frac{R^c}{\sum_{i=1}^p a_i})^T$. Fortunately, it's reasonable to expect this term to be very small compared to the first term $A^c_m b_k^T$ especially in the salient areas where the first term is large and contribute the most to the interpretability visualization. This is because original NMF prototypes $B^c$ are explicitly optimized to approximate the features $A^c$. Though we use the assumption that prototypes are strictly orthogonal to derive this conclusion, a similar logic also applies to not strictly orthogonal NMF prototypes, because NMF is exactly an optimization process to make the prototypes as diverse/orthogonal as possible, yet serving as a good basis to span a feature space covering the features of the target class.

We have accordingly revised the section 3.3 and 4.2 of the manuscript and A.2, A.3, A.4 of the appendix (after the reference section). Modified parts are colored blue.