B-cosification: Transforming Deep Neural Networks to be Inherently Interpretable

Thank you for your detailed review and valuable feedback. We appreciate your positive remarks on our work's significance and your constructive suggestions for improvement. We have carefully considered your comments and would like to address each point you raised.

• Inherent Interpretability of B-cos: Please note that B-cos explanations are not mere post-hoc explanations. Instead, the linear summaries W(x) of the models are in fact exact summaries (i.e., yielding the same outputs when applied to x) of the models’ computations and are optimized during training to align with task-relevant features, as described in [6], and have further been found to compare favorably to post-hoc explanations (Figure 6 in [6]). While we agree that the B-cos explanations of course constitute a simplification of the full network, previously reported results suggest that they indeed capture important aspects of the models’ computations, allowing them to e.g. localise class-specific features at a fine granularity. Our work builds on those prior findings and examines how to bring these previously reported benefits to pretrained conventional models.
• Model Randomization test performance: Note that the localisation scores give clear evidence that the explanations are highly model-dependent: if they were not, they would not allow for localizing specific classes with such high specificity (>90%, see Figure 6 in [6]). Further note that conversely, explanation methods that failed the randomisation tests, as proposed in Adebayo et al., notably score very low in the grid pointing game. We are happy to include results regarding the model randomisation tests in the final version of the paper. Lastly, as can be seen in the new figure we provide in the attached global response PDF (Figure 6, right), before fine-tuning the models for inherent interpretability, they give highly uninterpretable explanations (cf. ‘at start’).
• Multiple seeds: As we write in our global response, we now provide results over three runs corresponding to Tables 2, 3, and 4 in Tables 5, 6, and 7, respectively, with standard deviations. From tables 5-6, we find that the conclusions in Section 3.2.2 remain unchanged — the simplest approach of directly setting B=2 and removing the bias at the beginning performs as well as more involved strategies such as learning B or decaying bias. Also, from Table 7, we find that the results regarding classification and interpretability performance, as in Section 4.1, are supported by repeated experiments.
• Aggregation of CLIP vectors: We agree with the reviewer that the aggregation mechanism changes the model’s behavior and thus does not faithfully reflect the full model’s predictions. We introduce this aggregation mechanism to show that our B-cosification approach allows us to interpret intermediate features of the CLIP model at a high level of detail (clearly outperforming the localisation ability of both full models), as the features still maintain a higher level of spatial resolution. This highlights an additional aspect in which B-cosified models could help better understand the inner workings of foundational models.
• Speedups in fine-tuning: We find that the speedups are often much better than 2x for convolutional models (Table 4). We agree that the speedups are lower for ViTs, and believe this could be because our method could further be optimized for ViTs, e.g. see the discussion on ViT performance in response to reviewer DdYE.
• Interpolation: For aggregating and explaining the intermediate features of the CLIP model, many options exist - e.g., attention pooling as in the original (and B-cosified) CLIP model, average pooling, or max pooling. To better understand the implications of the feature aggregation, we propose a mechanism based on the cosine similarities that allow us to interpolate between average (p=1) and max pooling (p=inf); note that since both of them differ from attention pool (denoted by B-cos CLIP in the figure), neither of them give the same results as attention pool. These results also indicate that the attention pooling mechanism selects the embedding features in a more specific fashion than mere average pooling, but aggregates the features in a coarser fashion than selecting only the features with high cosine similarities (p$\gg$1).
• Qualitative examples: We provide additional randomly selected examples corresponding to both Figures 1 and 5b in Figure 7 of the attached global rebuttal PDF and will provide an extended set as a supplement to the final version of the paper.
• Performance on text modalities: We agree this would be a very interesting investigation. Since B-cos models were primarily used for image classification, we also focused on the same setting in this work. However, exploring how to extend to other modalities, such as text, would be a fruitful direction for future research.
• Which models are not B-cosifiable? Based on our experimental results, we believe this question does not have a yes or no answer - different models might benefit from a B-cosification to different degrees. For example, while the convolutional models we tested adhere closely to the original formulation of the B-cos models and thus lend themselves well to B-cosification with a high degree of localisation, our results regarding the feature aggregation mechanism (attention pooling vs. our proposed approach) suggest that some models might lack the necessary alignment pressure to exhibit equally strong localisation abilities. We appreciate the reviewer’s suggestion and will update our discussion accordingly.

I would like to remind the authors to publish their code.

The answer in the paper checklist says "... and [we] will make all our modifications available for ensuring full reproducibility of the reported results." Yet, the github repo https://github.com/shrebox/B-cosification remains empty.

My score was influenced by this belief, as I believe the additional usability of B-cos is a major positive point of this work. As such, I encourage the authors to adhere to the statements made during the submission period in a timely manner. Also, it's bad for distributing your work, as other (like me) won't use this work if there is no codebase available. Best regards

Thank you for your valuable feedback. We appreciate your recognition of the significance of our work and your constructive suggestions. We have carefully considered your comments and would like to address each of the points you raised.

• How do localization scores vary with epochs?: As we write in our global response, we now report these results in Figure 6 in the attached PDF. We find both quantitatively (left) and qualitatively (right) that the localization improves very quickly. For example, for ResNet-18, the localization improves from 21.5 to 82.0 after just one epoch of fine-tuning, which is close to 86.9 achieved by the B-cos model trained from scratch. On the other hand, modifications to the model architecture at the beginning of the B-cosification process lead to an initial drop in accuracy, from 69.8 for the standard model to 59.9 after one epoch. However, we recover the accuracy much quicker than training from scratch, with the B-cosified model needing just 29 epochs to reach the accuracy of a B-cos model trained from scratch for 90 epochs.
• How is bias handled? Apologies if this was not sufficiently clear. In the ablation studies, we remove the bias when varying B since that is the setup in B-cos as well as our B-cosification process (Table 3); for an additional discussion, also see ‘Completeness/bias’ in answer to reviewer Z7HS.
• Performance of B-cosified models in Tables 2-3: The baseline in Tables 2-3 refers to the standard and B-cos models trained for 90 epochs. During B-cosification, we start from the 90 epoch trained standard model and further fine-tune, which leads to performance gains. For a fairer comparison, we also report results when fine-tuning the baselines for an additional 90 epochs in Table 8 (blocks 3-4) in the global response PDF, and find that the B-cosified models perform similarly or better, while also being interpretable.
• Speedup epochs: In Table 4, we always fine tune for 90 epochs, however, the speedup column shows the point at which the B-cosified model reaches the performance of the corresponding B-cos model when trained from scratch. We now also report the epoch number at which this ‘overtake’ happens at in Table 8 ($t_\text{ovt}$) in global response PDF, e.g. for ResNet-18, we find that the B-cosified model reaches B-cos performance with just 29 epochs of fine-tuning.
• ViT performance: Please note that in comparison to many other post-hoc explanation methods, the localisation scores of both B-cos and B-cosified ViTs are relatively high, differing only by a few percentage points. We will analyse and discuss (e.g., impact of GELU) potential reasons for the differences between the ViT models in more detail in the final version.
• Missing reference: Thank you for pointing this out, we will correct the missing reference for the final version.

Thank you for the detailed feedback and the suggestions to improve clarity; we will incorporate them as well as the following answers in our revision.

• Full fine-tuning: The research question we examine is how to leverage existing pre-trained uninterpretable models to make training more interpretable ones more cost-effective. To endow pre-trained models with inherent interpretability, it is indeed inherently necessary to fine-tune the full models, as we aim to fundamentally change how they operate - however, we find that this approach is significantly cheaper than training the corresponding interpretable models from scratch (up to 9 times speed-up, Tab. 4).
• Multiple runs: See global response.
• Tab 4 small increments: The increments in accuracy are indeed small but also not our core focus. Instead, we show that it is possible to maintain the original models’ accuracy whilst significantly increasing their interpretability, at a significantly lower cost than when training from scratch as in [6].
• B-cos more interpretable: As reported in [6,7], B-cos explanations explain the full model from input to output and are significantly more localized, and allow to visualize the important features in color (Figs. 1,7 in [6]), while also allowing to understand misclassifications (Fig 9, [6]), making it easier for humans to see understand the model's predictions. Please note that in the examples in Fig. 1, the rows correspond to different models, and so are not directly comparable. I.e., the CLIP model might indeed use the grass to predict the sheep, while the B-cosified CLIP model might not.
• Comparing to Text2Concept: [28] shows that a linear transform can map an arbitrary model's features to CLIP features. Hence, training such a transform on an existing B-cos model serves as a good baseline for B-cosification. We follow [28] and use ImageNet to learn the linear map. However, other parameters are different: we use a B-cos encoder instead of a standard model, a CLIP ResNet-50 instead of ViT-B/16, and evaluate across a broader range of datasets (Fig. 4). Understanding the performance difference requires further investigation, but is orthogonal to our current work.
• Dynamic linearity: As discussed in prior work (cf. [18]), a linear layer followed by ReLU or MaxOut is ‘piece-wise linear’: for every input x, the prediction is effectively computed via a linear transformation of x, with the choice of the linear matrix W depending on x. Dynamic linearity (cf. [6, 7]) is a generalization of this concept, making the dependence of the linear matrices on the input more explicit. In particular, ‘dynamic linear models’ compute their output via an input-dependent linear transformation of the input y(x) = W(x) x. Finally, we agree with the reviewer that when bias terms are employed, “piecewise affine” is more accurate in contrast to commonly used nomenclature; given that the bias can be modeled via an additional input dimension, however, we opted to stick to standard nomenclature.
• Completeness/bias: The importance of biases has been discussed in prior work: [48] discusses how gradient-based explanations tend to neglect important contributions from biases (cf Eq. 1, [48]), which have been shown to play a significant role in the models’ outputs (Fig. 13 in [7]); completeness further constitutes one of the foundational axioms of IntGrad [49]. We will revise to better reflect these connections.
• Novelty of conversion: No, while [6] discussed and emphasized the differences between B-cos and conventional networks, we leverage the similarities and show that the existing weights of pre-trained DNNs can be used to efficiently obtain performant and interpretable DNNs, despite e.g. using a different number of input channels (3 vs. 6) and activation functions.
• Input encoding: No, we do not discard the first convolutional layer and train a new one from scratch; instead, as part of our contributions, we show how to construct a first layer and compute the corresponding weight that takes six channels as input and is nonetheless functionally equivalent to the original layer with no additional training.
• Activation functions: We keep ReLU since (i) we want to at first maintain functional equivalence with the original model, (ii) using ReLU does not affect the dynamic linearity of B-cos, (iii) the pre-trained weights were trained with them, so using ReLU keeps us closer to the original models.
• Normalized weights: Using normalized weights ensures that the output can be maximized only if the input and weight vectors align, i.e. if the cosine term in the dot product between them is high, see [6]. This helps interpretability, since weights being aligned to inputs better highlights the most important regions.
• Localization score: It refers to the GridPG metric used in [6,7].
• Robustness of GridPG: GridPG [6, 7] computes the ratio of attributions in a grid cell to the total attributions, and so is invariant to scaling To reduce the risk of spurious relationships, the metric uses grids where each image is of a distinct class and is classified correctly with high confidence by the original model, implying that it is unambiguous to the model.
• CLIP Interpretability: We show p=7 and p=19 for illustrative purposes, the full trend across p is shown in Fig 5a. We also provide a more detailed discussion on the CLIP experiments in our response to reviewer JC7z.
• Natural/Specialized/Structured data: We follow the CLIP benchmark [26] for zero-shot and linear probing results (Fig. 4) and categorize datasets based on the images they contain.
• Sec 3.1: We do mention it, see L139.
• "Outperform" performance measure: We report accuracy for image classification, and zero-shot performance for CLIP models.
• Citations: Thank you, we will add them.
• Eq1: We use row-wise dot products, following prior work (cf. Eq. 9 in [6]). We will revise for clarity.

Thank you for your constructive feedback on our work. We have carefully considered your comments and would like to address each point you raised.

• Limited novelty: To the best of our knowledge, we are the first to investigate how to transform existing uninterpretable models into inherently interpretable B-cos models. In particular, we find that we can effectively B-cosify existing models at a fraction of the cost of training them from scratch, which has the potential to make interpretable models far more accessible than they currently are. In order to be able to do this, we discuss in detail how the models differ and devise specific solutions to address those differences. In particular, we first show that it is possible to cast conventional models in the same framework as B-cos models without changing the function that they represent (i.e., we adapt the model $f_{org}$ to a B-cos version $f_{Bcos}$ with B=1 whilst maintaining functional equivalence, such that $f_{org}(x) = f_{Bcos}(x)$ $\forall$ $x$ ). Specifically, in this context we address the input encoding and normalization, activation functions, weight normalization, and incorporate the pre-trained weights. Thereafter, to increase interpretability, we increase the parameter B and decrease the models’ biases. This, of course, functionally changes the model and hence requires fine-tuning as a final step to adapt and recover the model’s performance to the new changes. We are unaware of previous work that explored this and would be grateful if the reviewer could point us to relevant related work that would limit the novelty of our contribution.
• Complexity of fine-tuning: We indeed train the models for a commonly used number of epochs (90) - however, to understand the efficiency gains by using pre-trained models, apart from accuracy and localisation scores, we also report the speed-up for reaching the same accuracy as randomly initialized B-cos models in the results (Table 4). We are very grateful for the reviewer’s suggestion to provide a visualization of accuracy and localisation scores over the training epochs, which we added in the attached PDF in Figure 6, as this helps clearly see how quickly the models’ explanations improve in their localisation (localisation scores > 80% after the first epoch). While the accuracy generally improves over the course of 90 epochs, we find that they tend to reach the performance of B-cos models trained from scratch very quickly (e.g. ResNet-18 reaches the performance of the original B-cos models within 29 epochs). This is our primary contribution.
• Fine-tuning base models: In Table 8 (column blocks 3-4) in the PDF attached, we report the results for the standard pre-trained and B-cos pre-trained models fine-tuned further for 90 epochs in the same way as proposed in the B-cosification process for two CNNs and two ViTs models. We find that even after fine-tuning the pre-trained models (standard and B-cos) further, B-cosified models perform competitively with conventional DNNs (column block 3) and outperform the B-cos DNNs (column block 4) consistent with the original findings mentioned in the main submission. We will provide full results in our revision.
• Accuracy of B-cos trained from scratch: In Table 8 (column 5) in the global response PDF, we now report this additional column for two CNNs and two ViTs models. We will provide full results in our revision.
• Writing fixes: Thank you for pointing these out; we will make the changes in our revision.