SPADE: Sparsity-Guided Debugging for Deep Neural Networks

We thank the reviewer for noting the experimental strengths of our method, as well as the intuitive design and presentation. We address the weaknesses raised by the reviewer below.

• Slow computation time/massive resource requirements to apply at scale. We thank the reviewer for raising this concern, which was also raised by others. To address this, we note that the bottleneck is the sparsification cost. Therefore, we ran additional experiments with a more efficient solver, to provide speed-accuracy tradeoffs. The results are described in the general response above, but in a nutshell, using the fastest pruning strategy drops the per-example requirements to 70 seconds on ResNet 50, at a small fidelity drop that still improves substantially over the baselines.

• Feasibility and fairness of calibration. We agree that the calibration can be infeasible. In that case, we propose a fixed sparsity schedule independent of interpretability method, that increases linearly with layer depth, and demonstrate that such a schedule is also quite effective (Appendix D/Table D.14). We also believe that our calibration does not unfairly leak information from the test data, as we use different data samples (also drawn from the ImageNet validation set) and different Trojan patches (emoji) for calibration.

• Comparison with Wong et al. We agree that our approach differs from Wong et al, in that we use per-example sparsity, whereas they train a single sparse FC layer for use in all examples, and for all interpretation methods. However, our approach is quite original (something that we see as a strength), and so Wong et al. was the closest approach available in the literature, thus, the best competitive baseline. We updated Section 4.1 to highlight the issues that you have raised.

• Novelty. We feel that our method is novel in the sense that it uses sample-specific, dynamic sparsity to improve model interpretations - so much so that we had trouble finding baselines to compare against, except for Wong et al, which as you note is an imperfect comparison. Regarding Panousis et al., we first note that ICCV was in October of this year, while the deadline for ICLR was in September; thus this is a concurrent work. Nevertheless, note that the method described in this paper creates sparse interpretable networks with a specific architecture (concept bottleneck), which is substantially different from what our method does - namely, provide a way to create better interpretations for dense models with a standard architecture.

Questions:

1. We show the sparsity ratios in the left panel of Figure 4 of the paper; generally, later layers have very high sparsity, while earlier layers have less sparsity.

2. The samples in the calibration sets were randomly sampled from the ImageNet validation set; we ensured that there was no intersection between these samples and the samples that we used for the evaluation.

3. SPADE traces could perhaps be stored as sparse matrices. But we do not foresee a use case where large numbers of SPADE traces would need to be stored, as these are just intermediate computations used to compute an interpretation. They could also be deterministically recreated by randomly generating and then storing the exact image transforms used to prune the network (most efficiently by fixing and storing the random seed).

4. Semantically similar images have a higher overlap in their sparse traces than do unrelated images. We added a figure to demonstrate this in Appendix J.

5. Our method can be applied to virtually any architecture, as e.g. convolutional layers can be unfolded to linear operators, on which SPADE is applied. We have illustrated this by showing results on ConvNEXT models, which are a hybrid between CNNs and Transformer-based architectures. We will try to add further results on this point in the next revision.

We thank the reviewer for their positive review, and for noting the clarity of our idea and the demonstrated effectiveness of our method.

As all reviewers noted that the computational requirements of SPADE can limit its practical use, we amended the paper to propose other variants of SPADE that provide different speed/accuracy tradeoffs. In particular, for a small accuracy drop, SPADE preprocessing can be done in 70 seconds per example.

We respond to the reviewer’s questions below.

1. W_sparse is meant to indicate that the argmin is taken over all weight matrices W that are sparse according to a preset sparsity level. We agree that this was not well explained and have adjusted the formula.

2. The re-calibration of the batchnorm was not applied to the model after SPADE preprocessing, as the model trace produced by SPADE is not useful for inference. The similarity result for which the batchnorm is applied was only presented as a sanity check that the top-1 prediction of the trace is still close to the network prediction.

3. Thank you for observing that we do not use rotations; we corrected this in the figure.

4. The image patches are contiguous regions of the image that, according to a low-resolution saliency map method (applied without SPADE) are the most important evidence for a specific possible predicted class. In our human evaluation, we used misclassified examples and produced image patches for the correct and predicted class, filtering only for those input images for which these do not intersect. The idea is that if the class visualization is human interpretable, the evaluator would be able to use the visual similarities between the class neuron visualization and the image patch to find the part of the image that resembles what the neuron is ‘looking for’.

5. Thank you for the correction, we will fix the typo.

We thank the reviewer for their thoughtful review, and for noting the quality of our presentation and evaluation. With regard to the weaknesses described by the reviewer, we address them below.

• Small-scale ImageNet experiments. The 140 test samples were chosen at random from the ImageNet validation dataset, subject to the following conditions: that the number of samples for each Trojan patch (emoji) was the same, and that only samples where the predicted label was changed by the Trojan were selected. The small number is due to the fact that each sample required custom pruning with SPADE, which takes ~40 minutes for each saliency method. We do believe that while the relatively small sample size makes the evaluation more noisy, it does present a useful comparison of the methods, especially as the results are quite consistent across datasets and saliency methods.

  However, due to your and others’ feedback, we will add to our paper experiments on a modified version of SPADE, that uses a more efficient pruner to create the saliency maps. Using this version of the method, we were able to run SPADE + LRP on 21121 samples from the ImageNet validation set (all that met our other criteria) in 120 GPU- hours on GeForce RTX 3090 GPUs.

• Non-negligible cost for constrained optimisation of sparse network. In the original work, we conceived of SPADE as a method for human debugging, for which perhaps a few dozen examples would be examined, and so we believed the per-example cost to be acceptable. However, as described in the previous answer, SPADE can also be used with more efficient sparsity solvers at a small accuracy cost (but still an improvement over the baselines), reducing the cost to 70 seconds per example for a ResNet50. We provide these tradeoffs below, and also in Appendix I of the revision.

• Limited novelty. We respectfully disagree with this assertion. Other works (with the exception of Wong et al (2021), which we address separately), construct sparse networks and demonstrate that they are somewhat more interpretable than dense networks, at a cost of some degradation in the performance of the network on the desired task, e.g. image classification. In particular, these networks are intended for inference, and replace the dense network entirely (in cases where the sparse network is obtained from a dense one, rather than trained from scratch). Conversely, the sparsification in SPADE is used as an additional step during the computation of an example interpretation, and does not produce a replacement network for inference or any other purpose. The sparsification in SPADE removes network connections that are least relevant to the particular example being studied. This preprocessing reduces noise in interpretability methods and increases their accuracy and usefulness on that example, but is not intended to produce networks useful for classification; nor do we propose using SPADE during regular inference. Note that this means that there is no inference performance tradeoff (in time or quality) to using SPADE, as SPADE is used solely for creating interpretations, and regular inference is not affected.

  The only other work to our knowledge that uses sparsity in this way - solely for the creation of better interpretations - is the work of Wong et al, which replaces the final layer with a sparse one. However, unlike that work, our method introduces the creation of example-specific traces through the network. This, to our knowledge, is completely novel, and we believe to be the core driver of the improved saliency map accuracy results.

• Fairness and fidelity of input saliency maps. We respectfully disagree with large parts of this point as well. We consider the preprocessing with SPADE to be a denoising step that is part of the saliency map creation, and so would argue that the saliency maps are still created with respect to the original, dense network. As network fidelity is always a concern with saliency maps, we validate this in the human study by using SPADE in the process of creating neuron visualizations and verifying that these are still relevant to the original network. We also performed an additional experiment (please see general comment above), which demonstrates that preprocessing with SPADE allows saliency methods to identify pixels that have a greater effect on the confidence of the dense network, providing additional evidence of our method’s applicability to the original, dense network.

  We do, however, note the reviewer’s excellent observation that combining interpretability methods with SPADE can lead to some randomness in the resulting saliency map, due to a small amount of randomness in the pruning algorithm. This can be overcome if necessary by randomly generating and then fixing the transforms used to create sample batches used for pruning.

Thank you for your clear explanation of the strengths and weaknesses of SPADE. We would like to address the weaknesses below.

Weakness #1: Transparency about the limitations of the proposed approach.

Thank you for your suggestion that we better address the limitations of our approach. We have updated the submission to more directly clarify these points, in particular addressing the case where ground truth is not available for any data, and so tuning is not possible, and provide results for a much faster implementation of SPADE, using a faster sparsity solver, reducing per-example computation time to 70 seconds.

To go through your points individually:

• Sparsity ratio tuning. We cross-validate by using a different set of emoji (and images) for tuning than we use for the main experiment. This is, of course, an imperfect proxy for the case where we are not looking for emoji trojans at all. Therefore, we also ran experiments where we used fixed sparsity ratios that are linearly interpolated by layer depth, from 0 in the initial layer to 99% in the final layer. We show in Table D13 that our method is also effective using these ratios.

• Computational cost. We acknowledge that the computational cost of using SPADE can be high when using the method on many examples, although we believe it is reasonable in cases where examples are created for human review, and so perhaps a few dozen are needed. Based on your and other reviewers’ feedback, we added a faster version of SPADE that requires 70 seconds per sample. Please see the general response to the reviewers above, and appendix I of the revision for details.

• Strength of results. We would push back on the assertion that the results are not that strong. Preprocessing with SPADE improves the attribution map AUC for all methods tried. The gains are certainly larger where the accuracy gap is higher, and smaller where the accuracy is already very high, and so there is not much room for improvement. Concretely, the prior work of Wong et al., ICML 2021 improved attribution AUC on average by 0.5%, whereas SPADE improves results by 4.2-5.5% on average (depending on the variant used).

  We are not aware of any research that shows that the Trojan emoji evaluation favors any specific methods, and so have no reason to believe that gains from SPADE would be greater in ‘real-world’ scenarios. Specifically, we emphasize that SPADE largely closes the gap between different saliency attribution methods, allowing the field practitioner to choose one that is best suited for their task on other criteria.

Weakness #2: Clarity about experimental design

Thank you for your very concrete and helpful suggestions on improving the text. We will update the draft to better clarify these points.

To summarize our answers here:

• Clarify motivation for the Trojan experiment setup. Our motivation for using them in this paper is that, to our knowledge, Trojan patches are the best available way to evaluate saliency map accuracy, which was our motivation for using them. We agree that our method cannot be used to find backdoors that are not present in the test set; we will add this limitation to the paper.

• Saliency map experiments. We calculate accuracy only for test samples with Trojan examples, as those are the ones for which there is a ground truth to compare to. As the Trojan patches are not 100% effective for changing the label, we only use examples in which the predicted label (of the original network) is changed due to the presence of the Trojan. We always compute gradients (or any other saliency metric) with respect to the predicted (Trojan) class for the experiment, as that is the one for which the ground truth is available. However, SPADE can be used to compute saliency for any possible class as well.

• User study. An “image region” is a section of an image that ScoreCAM, a broad-resolution saliency method (without SPADE preprocessing), selects as the most highly relevant for a predicted class label. While it is true that this method for creating regions is imperfect and possibly noisy, due to the very concerns regarding saliency attribution methods that our paper is addressing, we believe that this is the best proxy for the true model behavior that was possible to use. We compute these regions for two classes for each model (so one region is the ‘best’ evidence for Class A and the other region is the ‘best’ evidence for Class B) and require that the two not intersect. The neuron visualization (but not the name or any other description) of one of the two classes is then shown to the human evaluator, who is then asked to choose the correct patch for that class. Screenshots of this flow are provided in Appendix H.

Thanks for your response and suggestions.

User Study: We agree with the recommendation to avoid using the term "ground truth". Although ScoreCAM has been shown to be accurate, it is not perfect. Therefore, we have updated our manuscript, and replaced "ground truth" with your suggested term "alignment with ScoreCAM".

Question: Yes, "choosing to make a decision" refers to instances where the user did not select the "Can't decide" option.

Agreement: Yes, there are cases where users see the same image. We calculated the number of response pairs where users viewed the same image and did not select the "Can't decide" option. We then determined how many of these pairs resulted in the same decision. The results indicate that using SPADE leads to higher agreement among users.

• Relevance to the dense model. We would like to clarify that SPADE is intended as a preprocessing step for obtaining a network interpretation (such as a saliency map or neuron visualization). As such, the interpretation applies to the dense model, as the sparsified model aims to specifically fit layer-wise outputs, dynamically on the specific sample. By contrast, prior methods such as Wong et al. create a static sparsity mask, which is applied to all examples, but requires model retraining.

  In the submission, we verified this relevance claim in the human evaluation, by asking the evaluators to use the neuron visualizations obtained with SPADE to reason about dense model behavior (section 4.2.2).

  To further address this concern, we present an additional validation, using the Insertion/Deletion metric, defined as follows. For the insertion metric, we start with a blank image then replace the pixels with those of the original image in decreasing order of saliency (as a proxy for importance). With each pixel addition, we plot the confidence of the (dense) model in the predicted class; the final score is the area under the resulting curve, normalized by the model’s confidence on the full image. The deletion score is the converse - pixels are replaced with a default value in increasing order of importance, and normalized AUC is computed as before (in this case, a lower AUC is better, as it shows that more useful pixels were removed earlier).

  The results of this experiment are presented below. We observe that for both metrics, for 9 out of 10 saliency map prediction methods studied (average AUC improvement of 8.77 for the insertion test), preprocessing with SPADE allows saliency map predictors to select pixels to add/remove that have a greater impact on the confidence of the dense model, suggesting that preprocessing with SPADE improves the fidelity of the saliency maps to the dense model. This experiment has been added to the revised draft as Appendix J.