Smoothed Differentiation Efficiently Mitigates Shattered Gradients in Explanations

We thank the reviewer for their time and suggestions.

Before we address the feedback, we want to highlight that due to a typo, Fig. 1b visualized SmoothGrad with the wrong sample size. We caught this error while working on the supplementary material and included an updated version in Appendix I.

Weaknesses

Smoothgrad is hardly the SOTA method at this time. Therefore, the authors should show that Smoothgrad adds value compared with SOTA methods. For CNNs, comparisons could be provided with GradCAM

There has been a number of different sanity tests and evaluation metrics for attribution methods. Fidel-flipping seems to be arbitrary selected and does not provide a comprehensive evaluation.

We expanded our quantitative evaluation to include nine XAI methods (SmoothDiff, SmoothGrad, Gradient, GradientShap, GradCAM, Integrated Gradients, Input x Gradient, two LRP composites) and six evaluation metrics. We selected the five metrics proposed by Bommer et al. (2024), as well as the Pointing Game:

• Robustness: Avg-Sensitivity (Yeh at el., 2019), measures the change of attributions under slight input perturbations.
• Faithfulness: Faithfulness Correlation (Bhatt et al., 2020), flips of a random subset of pixels and computes the Pearson correlation coefficient between the predicted logits and average attribution for the flipped features.
• Localization: Relevance Rank Accuracy (Arras et al., 2021), measures the ratio of high attribution pixels within a provided ground truth mask.
• Complexity: Sparseness (Chalasani et al., 2020), computes the Gini index applied to the absolute values of attributions, indicating that an attribution consists of only a few important features.
• Randomization: Random Logit Metric (Sixt et al., 2020), computes the distance between an attribution and the attribution of a randomly selected non-target class.
• Pointing Game: (Zhang et al., 2018), check whether the point of maximal attribution lies within a provided ground truth mask.

Reference implementations from Captum (Kokhlikyan et al., 2022) are used for all XAI methods except LRP, for which composites from Zennit (Anders et al., 2021) are used. Metrics are evaluated on a pre-trained ResNet-18 using the the Quantus toolkit (Hedström et al., 2024) with default parameters. For SmoothDiff, SmoothGrad, Integrated Gradients, and GradientShap, attributions are computed using 20 samples.

Since this year's NeurIPS rebuttal policies prohibit the inclusion of PDFs, images and URLs, we provide our results in form of tables. To improve the readability of our results, we report the ranking of a method within each metric, followed by the mean score in parenthesis. With the exception of the Robustness metric, a higher score is better.

Table R1: Results on ResNet-18

On ResNet-18, SmoothDiff has the highest Robustness w.r.t. input perturbations, while the high Faithfulness score mirrors the favorable pixel-flipping performance reported in Section 4.4. The Randomization score demonstrates that SmoothDiff attributions strongly depend on the selected output class. SmoothDiff performs well in the Pointing Game and the Localization metric, both of which measure the localization of attributions within ground truth masks. With the exception of Sparseness, SmoothDiff places in the top 3 on all metrics, while being easy to implement. Since attributions qualitatively look sparse, we will further investigate this metric for the camera-ready version.

On VGG19 (see Table R2), SmoothDiff performs similarly well, showing that these results are not model dependant.

Table R2: Results on VGG19

There is no evaluations on modern neural network architectures such as ViTs

As we explain in Section 3.1, the performance of SmoothDiff highly depends on the selected Jacobian factorization. While SmoothDiff can be applied to arbitrary models, including ViTs, we have not yet found a factorization for which SmoothDiff outperforms SmoothGrad. We think that SmoothDiff's quantitative results on CNN (see Tables R1, R2) stand on their own, especially in light of how easy the method is to implement. We will update the "Limitations" section of our camera-ready version to explicitly mention the current lack of performant vector-Jacobian products (VEJPs) for ViTs.

Something we have yet to investigate is whether a partial application of SmoothDiff improves attributions over "vanilla" gradients. To provide an example in the context of ViTs, VEJPs could only be implemented on GeLU, computing "vanilla" gradients on all other non-linear layers.

The methodology seems to have been written with an unnecessary amount of math. It is very heavy to read and not easy to follow.

The first five equations are essential to the paper: Eq. 1-3 introduce the preliminaries SmoothDiff builds upon (Gaussian convolutions, SmoothGrad, Jacobian factorizations and vector-Jacobian products) and in Eq. 4-5, we propose SmoothDiff itself.

The remaining four equations are meant to serve as illustrative practical examples: Eq. 6 shows the Jacobian factorization on a Dense layer and Eq. 7-9 show the derivation of a VEJP on a ReLU.

Questions

Figure 2 shows that you can reduce the number of samples from 50 to 10. This would give you a speed up of 5X.

Figure 2 shows SmoothDiff's 5X increase in sample efficiency. In addition, SmoothDiff has a purely computational speedup of 2X at equal sample sizes, as shown in Section 4.3, Figure 4.

How do you get to two-three orders of speedup in the abstract?

The computational speedup of two orders is described in Section 3.3 "Computational performance", lines 187-194: Once VEJPs have been estimated in $n$ forward-passes, SmoothDiff attributions can be computed in a single backward-pass. Computing multi-class attributions for $k$ classes (or hidden neurons / concepts) only takes $k$ backward-passes, for a total of $n+k$ forward- and backward passes. SmoothGrad on the other hand requires $n$ forward- and $n$ backward-passes for each attribution, resulting in a total of $2nk$ passes through the model.

Assuming an attribution of all $k=1000$ ImageNet classes ($k$ could be even higher for concept attributions) using $n=50$ samples, SmoothGrad requires $2nk=10^5$ passes through the model, whereas SmoothDiff only requires $n+k=1050$. This corresponds to a purely computational speedup of two orders of magnitude, ignoring the increased sample efficiency.

Can the method be applied to a broad range of computer vision models?

Yes, SmoothDiff can be applied to arbitrary differentiable models, including computer vision models. For any function (e.g. a layer), a "naive" VEJP can be implemented that uses automatic differentiation (AD) to compute the full Jacobian on each forward-pass, keeping track of a running mean. However, for large and dense Jacobians, this "naive" approach requires a lot of memory and compute, negating SmoothDiff's performance benefits over SmoothGrad.

As we discuss in Section 3.1, the performance of SmoothDiff therefore strongly depends on the factorization. The reported computational speed-ups of 2X hold for both VGG and ResNet models. For the camera-ready version, we will update Subsection 3.2 to include the naive running mean VEJP and the "Limitations" sections to explicitly mention the current lack of performant factorizations for ViT.

Do the speed-ups hold for a broad range of attribution tests?

SmoothDiff's speed-ups (both computational and in sample efficiency) hold independent of evaluation metrics for attributions.

Thanks for your review and the raised points, which I found quite insightful.

Could you kindly elaborate on what are the SOTA methods that would be good to compare against? As far as I know, LRP or Integrated Gradients that the authors already compare to in Tables R1&R2, are among the stronger attribution methods. And it seems that SmoothDiff raises SmoothGrad to be ranked 1st among some of the metrics (e.g. Pointing Game). I have asked the authors to maybe compare with inherently-interpretable models (e.g. x-DNNs or B-cos) as a sort of "upper bound" and hard baselines. But I'm curious if you find any crucial attribution method missing from Tables R1&R2.

Thanks!

We thank the reviewer for engaging with our rebuttal.

I appreciate the updated experimental results section. The paper clearly shows that the proposed framework is an improvement over SmoothGrad.

Our results show that on ResNets, SmoothDiff not only outperforms SmoothGrad, but places in the top 3 in five of the six evaluated metrics, while being easy to implement.

In their initial feedback, the reviewer asked for a comparison with GradCAM as the SOTA method on CNN:

Smoothgrad is hardly the SOTA method at this time. Therefore, the authors should show that Smoothgrad adds value compared with SOTA methods. For CNNs, comparisons could be provided with GradCAM

We added this requested comparison with GradCAM in our initial rebuttal. Our results on VGG19 show that the proposed SmoothDiff method outperforms GradCAM in terms of Robustness, Faithfulness, Localization and draws first place in the Pointing Game.

However, SmoothGrad is not a SOTA method for attributions anymore, which is evidenced by the results that have been provided in the rebuttal.

We reemphasize that SmoothDiff not only outperforms SmoothGrad, but also state-of-the art LRP composites and GradCAM (the proposed SOTA method on CNN) on several metrics.

We agree with the sentiment expressed by reviewer "Meht" and kindly ask reviewer "WSGh" to elaborate on the additional SOTA methods they would like to see evaluated.

Also, It would have been good to include perturbation games such as Insertion and Deletion in the evaluation, which are very common technique for evaluating attribution methods.

We provide several evaluation metrics based on perturbations:

• Section 4.4 of our paper includes detailed pixel-flipping results based on the novel symmetric relevance gain measure introduced by Bluecher et al. (2024), which removes the influence of infilling and in-painting methods (i.e. insertions and deletions) on pixel-flipping.
• The Robustness metric Avg-Sensitivity (Yeh at el., 2019) measures the change of attributions under slight input perturbations.
• Faithfulness Correlation (Bhatt et al., 2020) flips a random subset of pixels and computes the Pearson correlation coefficient between the predicted logits and the average attribution for the flipped features.

We are willing to extend our evaluation to further perturbation-based metrics if the reviewer elaborates on which metrics they think are missing.

We thank the reviewer for their time and expertise, as well as for underlining the quality of our approach.

Before addressing the feedback, we want to highlight that due to a typo, Fig. 1b visualized SmoothGrad with the wrong sample size. We caught this error while working on the supplementary material and included an updated version in Appendix I.

Weaknesses

(a) Writing

I would say the paper is mostly well-written, as the paragraphs are easy to follow. However, the overall structure is poor [...] Basically from Lines 31 to 82, all of this should be spread to the Related Work section or preliminaries in the Method section.

Since this year's NeurIPS rebuttal policies prohibit the inclusion of PDFs, URLs and images, we will address these issues in the camera-ready version of the paper. We will restructure the "Introduction" and "Related Work" sections, provide more detail on both the problem setup and desired properties (see (b) below) and add a subsection for preliminaries.

The captions on the figures are also very uninformative.

We will provide more detailed captions in the camera-ready version of the paper.

I also think there is a formatting issue with the citations not having numbers

We have fixed this issue.

(b) Quantitative Evaluation of Attributions

I appreciate the author including the pixel flipping study. But by now there are many more ways of evaluating the attributions, and I don’t see why the paper should confine itself to only one. For example, [1] proposes Grid-Pointing Game, specifically targeting the localization and interpretability of the explanations.

We appreciate the suggestion and pointers to relevant metrics. Within the time frame of the rebuttal, we have opted to use the Quantus toolkit (Hedström et al., 2024), which provides a common interface for 35+ metrics, including the proposed pointing game. We additionally evaluated the five metrics proposed by Bommer et al. (2024):

• Robustness: Avg-Sensitivity (Yeh at el., 2019), measures the change of attributions under slight input perturbations.
• Faithfulness: Faithfulness Correlation (Bhatt et al., 2020), flips of a random subset of pixels and computes the Pearson correlation coefficient between the predicted logits and average attribution for the flipped features.
• Localization: Relevance Rank Accuracy (Arras et al., 2021), measures the ratio of high attribution pixels within a provided ground truth mask.
• Complexity: Sparseness (Chalasani et al., 2020), computes the Gini index applied to the absolute values of attributions, indicating that an attribution consists of only a few important features.
• Randomization: Random Logit Metric (Sixt et al., 2020), computes the distance between an attribution and the attribution of a randomly selected non-target class.

Captum (Kokhlikyan et al., 2022) is used for all XAI methods except LRP, for which Zennit (Anders et al., 2021) is used. Metrics are evaluated on a pre-trained ResNet-18 using default parameters. For SmoothDiff, SmoothGrad, Integrated Gradients, and GradientShap, attributions are computed using 20 samples. To improve the readability of our results, we report the ranking of a method within each metric, followed by the mean score in parenthesis. With the exception of the Robustness metric, a higher score is better.

SmoothDiff (SD) has the highest Robustness w.r.t. input perturbations, while the high Faithfulness score mirrors the favorable pixel-flipping performance reported in Section 4.4. The Randomization score demonstrates that SD attributions strongly depend on the selected output class. With the exception of Sparseness, SD places in the top 3 on all metrics, while being easy to implement. Since attributions qualitatively look sparse, we will further investigate this for the camera-ready version.

I expect Grid-Pointing Game score to be much higher than baselines, and this would further ensure that the qualitative observation (of less-noise for same # samples) indeed is supported quantitatively.

The results indeed show that SD performs well in the Pointing Game and the Localization metric, both of which measure the localization of attributions within ground truth masks.

[2] proposes a new way of evaluating attribution methods. [3] proposes a synthetic benchmark for faithfulness of such methods, or [4] proposes sanity checks for faithfulness.

While those benchmarks fell outside of the time scope of this rebuttal, we will include references to these metrics in the camera-ready.

(c) Baseline Attribution Methods

it is necessary to put the proposed attribution method in comparison with more attributions. E.g. LRP, Integrated Gradients.

We expanded our quantitative evaluation to include 9 methods (SD, SmoothGrad, Gradient, GradientShap, GradCAM, Integrated Gradients, IxG, two LRP composites), 2 models (VGG19, ResNet-18) and 6 metrics. The results on ResNet-18 are shown in the table above and results on VGG19 can be provided during the reviewer-author discussion (due to the character limit).

I also was not able to follow the discussion on Lines 186-186. Is SmoothDiff orthogonal to LRP? How would LRP benefit from SmoothDiff exactly?

LRP is computationally very efficient, computing attributions in a single backward-pass through the model. We argue that the methods enumerated in L183-185 are enabled by LRP's computational efficiency and would not be feasible using SmoothGrad. As an example, computing attributions w.r.t. $k$ hidden neurons in Concept Relevance Propagation (Achtibat et al., 2023) only requires $k$ backward-passes, whereas SmoothGrad with $n$ samples requires $nk$ passes.

SD is also computationally very efficient: Once the vector-expected-Jacobian product (VEJP) operator has been estimated in $n$ forward-passes, attributions can be computed in a single backward-pass. Computing attributions for $k$ classes / hidden neurons / concepts only takes $k$ backward-passes, like LRP.

We will clarify this in the camera-ready, as it is a key strength of SmoothDiff.

(d) Scope

There are many commonly used architectures that are significantly different from this paper. We of course have ViTs, with multi-head self-attention and softmax, ConvNext with GeLU instead of ReLU and many more.

SD can be applied to arbitrary models and factorizations, but the performance will vary.

For any function (e.g. a layer), a "naive" VEJP can be implemented that uses automatic differentiation (AD) to compute the full Jacobian on each forward-pass, keeping track of a running mean. For a GeLU $f(x)=x \Phi(x)$, this approach is very performant, only requiring $n$ evaluations of $f'(x)=x \Phi'(x) + \Phi(x)$, one for each sample. However, for very large and dense Jacobians (e.g. attention blocks in ViT), this "naive" approach requires a lot of memory and compute, negating SD's performance benefits over SmoothGrad.

Something we have yet to investigate is whether a partial application of SD improves attributions over "vanilla" Gradients. To provide an example in the context of ViT, VEJP could only be implemented on GeLU, computing "vanilla" gradients on all other non-linear layers.

To me it is acceptable if the proposed approach does not readily extend to this, but the lack of discussion on this matter concerns me.

We fully agree: for the camera-ready version, we will update the section on "Factorization" to include the naive running mean VEJP and the "Limitations" sections to explicitly mention the lack of performant VEJP for ViT.

Questions

Could the authors clarify if the final attribution map would be differentiable, and whether this also is the case for the compared baseline (SmoothGrad)?

This is a great question. Differentiating the gradient of a model would result in its Hessian, which describes the curvature of the model at a given input. Deep ReLU networks are piece-wise linear and have no curvature. Their gradients are piecewise constant and their Hessians therefore zero. However, by applying a Gaussian convolution to the gradient, curvature is introduced (see Fig. 1a). Mathematically speaking, the function obtained by applying a convolutional operator to the gradient function is therefore differentiable (with non-zero derivatives).

Implementation-wise, Hessians are usually computed via AD by nesting forward- and reverse-mode to evaluate a series of Hessian-vector products (see Dagréou et al., 2024). As we discuss in the paper, SD can be seen as a modified reverse-mode AD pass. It could therefore potentially be used within the computation of an "smoothed" HVP. While we don't feel confident enough to make claims about the practical differentiability of SD (outside of numerical approximations like finite differences), we will update the outlook of the camera-ready to mention 2nd-order explanations.

Limitations

There are more limitations in terms of applicability to currently used models (see concern (d) above). These should also be discussed in the limitations.

We have addressed this in our rebuttal of concern (d) and will update the limitations section of our camera-ready version.

I would like to thank the authors for their detailed and clear response. I'm mostly convinced that the promised revision of the manuscript will be stronger. There are however three more questions I would like to ask.

a) Could the authors however elaborate on the Localization Metric? It is slightly strange to me that the Integrated Gradients is worse than vanilla gradient. Could you elaborate on how many samples was used for Integrated Gradients? Also is there any random baseline number for this metric?

b) Additionally, I am generally against comparing attribution methods with human-annotations, as it is not guaranteed that the model should be using what we as humans expect. If the authors could provide an evaluation using GridPG [1,2], which creates a 2x2 grid of images with different labels and measures how much of the attribution of a class lies in the correct cell, which is therefore annotation-free, that would be even more convincing to me. I checked and there are multiple implementations of this available [3,4]. I understand that the time is limited, but this would strengthen the evaluation in my opinion.

If their results are compared to model-inherent methods such as x-DNN or B-cos, which are more recent, then this would also partially address Weakness 1 raised by Reviewer WSGh. I don't necessarily expect this method to beat model-inherent attributions, but a comparison to such methods would put this method on the (more recent) grid.

c) Lastly, have the authors done any experiments on a multi-label dataset, or simply images that have multiple classes, to see whether the multi-class attributions are valid? For example, an image with Person + Bike from Pascal VOC dataset, where the attribution for Person and Bike are estimated? I think this would help the paper, even if done only on a qualitative level, given that the goal is to estimate the Jacobian more efficiently. One can also do this with an ImageNet model, just using images that have more than one category in them.

[1] CoDA Nets, CVPR 2021, https://arxiv.org/abs/2104.00032

[2] B-cos Alignment for Inherently Interpretable CNNs and Vision Transformers, https://arxiv.org/abs/2306.10898

[3] https://github.com/B-cos/B-cos-v2/blob/main/interpretability/analyses/localisation.py

[4] https://github.com/sukrutrao/Attribution-Evaluation/blob/master/scripts/evaluate_localization.py

We want to thank the reviewer for their thorough feedback and are pleased to see the strengths of our method being recognized.

Before we address the feedback, we want to highlight that due to a typo, Fig. 1b visualized SmoothGrad with the wrong sample size. We caught this error while working on the supplementary material and included an updated version in Appendix I.

Weaknesses

However, this contribution is ultimately limited to improving the approximation of smoothgrad, and not necessarily to improving the reliability or fidelity of attributions under challenging model conditions.

At its core, SmoothDiff efficiently approximates arbitrary integrals over gradients and Jacobians. Within the scope of our paper, we look at only one type of integral: the Gaussian convolution on which SmoothGrad is based. As we mention in our outlook, more complex integrals could be investigated, e.g. convolutions that take the geometry of the data manifold into account to improve reliability.

We also want to emphasize that both SmoothDiff and SmoothGrad are robust and reliable methods. To demonstrate these properties, we have we expanded our quantitative evaluation to include five metrics proposed by Bommer et al. (2024), as well as the Pointing Game:

• Robustness: Avg-Sensitivity (Yeh at el., 2019), measures the change of attributions under slight input perturbations.
• Faithfulness: Faithfulness Correlation (Bhatt et al., 2020), flips of a random subset of pixels and computes the Pearson correlation coefficient between the predicted logits and average attribution for the flipped features.
• Localization: Relevance Rank Accuracy (Arras et al., 2021), measures the ratio of high attribution pixels within a provided ground truth mask.
• Complexity: Sparseness (Chalasani et al., 2020), computes the Gini index applied to the absolute values of attributions, indicating that an attribution consists of only a few important features.
• Randomization: Random Logit Metric (Sixt et al., 2020), computes the distance between an attribution and the attribution of a randomly selected non-target class.
• Pointing Game: (Zhang et al., 2018), checks whether the point of maximal attribution lies within a provided ground truth mask.

Reference implementations from Captum (Kokhlikyan et al., 2022) are used for all XAI methods except LRP, for which composites from Zennit (Anders et al., 2021) are used. Metrics are evaluated on a pre-trained ResNet-18 using the the Quantus toolkit (Hedström et al., 2024) with default parameters. For SmoothDiff, SmoothGrad, Integrated Gradients, and GradientShap, attributions are computed using 20 samples.

Since this year's NeurIPS rebuttal policies prohibit the inclusion of PDFs, images and URLs, we provide our results in form of a table. To improve the readability of our results, we report the ranking of a method within each metric, followed by the mean score in parenthesis. With the exception of the Robustness metric, a higher score is better.

Table R1: Results on ResNet-18

On ResNet-18, SmoothDiff has the highest Robustness w.r.t. input perturbations, while the high Faithfulness score mirrors the favorable pixel-flipping performance reported in Section 4.4. The Randomization score demonstrates that SmoothDiff attributions strongly depend on the selected output class. SmoothDiff performs well in the Pointing Game and the Localization metric, both of which measure the localization of attributions within ground truth masks. With the exception of Sparseness, SmoothDiff places in the top 3 on all metrics, while being easy to implement.

Questions

The method approximates the expected Jacobian by ignoring cross-covariances. Can you quantify how this approximation affects attribution fidelity in practice, especially for deeper or more nonlinear architectures?

We have added results on the ResNet-18 architecture in the section above, demonstrating that SmoothDiff's properties of Robustness, Faithfulness, Localization and Randomization not only hold for sequential models (we can provide a similar results table for VGG19), but also on models with branching residual connections (see Table R1).

We have also experimentally investigated ignored cross-covariances on a pre-trained VGG19 model. For this purpose, we compared SmoothDiff and SmoothGrad attributions (the "smoothed gradients"), by flattening them into vectors:

• In terms of cosine similarity, the directions of attribution vectors align well.
• In terms of magnitude, attribution vectors differ.
• The difference vector between attributions (which is caused by neglected cross-covariance and random sampling) therefore aligns well with the attributions

During heatmapping, individual features within an attribution are normalized, such that heatmaps only depend on the direction of an attribution vector. Due to the directional alignment of the difference vector, we therefore observe that neglected cross-covariances don't strongly affect heatmaps, which is in line with the SSIM convergence plot in Figure 2.

We will address cross-covariances in the camera-ready version by discussing these results and by adding a plot showing the convergence of attributions in terms of cosine similarity.

Limitations

Missing Engagement with Attribution Reliability Literature The paper does not sufficiently engage with foundational and recent research that calls into question the reliability of gradient-based attribution methods under certain conditions.

As mentioned above, we have added evaluation metrics according to recommendations by Bommer et al. (2024), notably Avg-Sensitivity (Yeh at el., 2019), Faithfulness Correlation (Bhatt et al., 2020), Relevance Rank Accuracy (Arras et al., 2021), Sparseness (Chalasani et al., 2020), the Random Logit Metric (Sixt et al., 2020) and the Pointing Game (Zhang et al., 2018).

• Hooker et al. (2018) showed that many attribution methods, including SmoothGrad, can perform no better than random baselines unless the model satisfies specific invariance properties.
• Ilyas et al. (2019) emphasized that the problem is often not the attribution method, but the model itself, which may rely on “non-robust features” imperceptible to humans.
• Shah et al. (2021) and Srinivas et al. (2023) demonstrated that models trained for adversarial or off-manifold robustness yield much more faithful gradient-based attributions.

We want to thank the reviewers for these detailed pointers.

• Hooker et al. find that "SmoothGrad-Squared" (SG-SQ), a variant of SmoothGrad outperforms both SmoothGrad and other methods on their proposed ROAR benchmark. We will investigate whether an analogous "SmoothDiff-Squared" is viable for the camera-ready.
• While we have yet to investigate SmoothDiff's behavior w.r.t. the "non-robust features" described by Ilyas et al., its performance in the Robustness metric of Avg-Sensitivity (Yeh at el., 2019) in Table R1 shows promising results.
• We provide a comparison with $\beta$-Smoothing (Dombrowski et al., 2019) in Appendix E, which was specifically proposed as a method with off-manifold robustness. We demonstrate that SmoothDiff outperforms $\beta$-Smoothing quantitatively in Table 1 and qualitatively in Figure 7.

[The authors should] clarify that their improvements pertain to a fixed-model regime, i.e., when the model is frozen and cannot be retrained for interpretability.

We will clarify the fixed-model regime in the camera-ready version of our paper.