One Wave To Explain Them All: A Unifying Perspective On Feature Attribution

We first would like to thank reviewer UvKx for the reviews on our work and underlining the potential of our idea, regarded as innovative. We would like to address the comments raised by the reviewer on our work. These comments will be taken into account in our work and will help us improve the quality and the clarity of our work.

Interpretability of wavelet coefficients

The reviewer challenged our assertion that wavelet coefficients are interpretable coefficients and stated that we did not provide enough justification. We kindly refer the reviewer to section 2.2, page 3, where we explain why wavelets are interpretable features for feature attribution. We add the wavelet transform of a signal provides a more interpretable representation of this signal than raw pixels or Fourier transforms by capturing spatial and frequency information. Fourier transforms only encode frequency content whereas wavelets preserve localization, making them particularly effective for analyzing structured data like audio and images. In audio, for instance, wavelets isolate transient components corresponding to phonemes, yielding a more intuitive representation of meaningful patterns. Wavelets extract high-level features that better convey the overall structure of a signal, making them a more interpretable alternative to the original data representation. For images, wavelet coefficients can be intuitively understood by viewing them as capturing details at different scales, like how our vision works when we look at an image from different distances. Wavelet coefficients align with the definition of interpretability by Kim et al. (2016, [1]), as the decomposition into different scales enables users to predict how the method will emphasize on specific features (e.g. edges or gridded patterns on images). We would like to thank the reviewer for pointing this out and will further enrich section 2.2. with the provided expanation.

On the scope and positioning of the paper and our claim on unification

The reviewer stated that our paper could be situated amond approaches that improve model performance or alternative forms of analysis. While this perspective is interesting, as our work indeed bridges the gap between explainability and model robustness for instance, we believe that our work aligns well with explainability and feature attribution as we essentially propose to attribute an importance score to input features, taken as the wavelet coefficients of the signal of interest rather than its pixels as in traditional feature attribution methods.

The reviewer mentioned that where our identification lies was unclear, a concern shared by R1. We thank the reviewer for this remark and acknowledge that indeed, we remain elusive on what we aimed at unifying. Unlike SHAP, we do not unify attribution methods but rather unify the domain in which attribution is carried out in a single input representation, the wavelet transform of this signal. We edited our manuscript to reflect the changes.

Clarifications on the evaluation of $\partial f_c(W^{-1}(z))/\partial z$

The reviewer mentioned that it was unclear how the computation of $\partial f_c(W^{-1}(z))/\partial z$ was achieved. In Pytorch, we require the gradients on the wavelet transform of the signal $z$ and reconstruct the original image from its wavelet coefficients, $x=W^{-1}(z)$. We carry out a forward and compute the derivative of the prediction with respect to the wavelet coefficients. We then re-express the signal into its wavelet transform to retrieve the gradients of the model with respect to the wavelet coefficients of the input signal. The novelty lies in expanding existing attribution methods (SmoothGrad and IntegratedGrad) into the wavelet domain.

Discussion of the faithfulness metric & additional references

The reviewer highlighted an additional definition of the faithfulness, as discussed in "Towards Faithful Model Explanation in NLP: A Survey". We thank the reviewer for this reference and will add it to our manuscript. Regarding our definition of the faithfulness, we acknowledge that there are several definitions of this notion. For this work, we chose the definition of Muzellec et al (2023), but as we believe that one metric is not sufficient to reflect the behavior of a method, we extensively evaluated our method with alternative metrics in the supplementary materials, section B.2, p.16.

The reviewer underlines that we do not discuss references from NLP. We would like to recall that since our method cannot be theoretically applied to text data, we did not considered works in this field (see Introduction, second column, l47-48), although we acknowledge that there have been many works in the field of XAI for NLP. We thank the reviewer for the reference and will add it to our manuscript.

• [1] Kim et al, 2016. “Examples Are Not Enough, Learn to Criticize! Criticism for Interpretability.” NIPS’16

We thank reviewer EDJ8 for the review and the comments on our work. We also thank the reviewer for pointing the duplicate reference, which we have corrected.

Regarding the evaluation criteria

The Reviewer said to include localization evaluation, Pointing Game, and visual explanation accuracy alongside faithfulness evaluation for assessing explanation methods for image inputs. We thank the reviewer for providing additional benchmarks. We kindly refer the reviewer to the supplementary material, as we have evaluated our method accross a wide range of common metrics from the existing literature. To complement these results, we are currently implementing the Pointing Game and Visual Explanation Accurary, as suggested by the reviewer, and will update the rebuttal if we get results.

Regarding the multiclass classification (Cats and Dogs examples)

The reviewer outlined the fact that from appendix A3, the distinction between dog and cat as target classes is less apparent in the wavelet domain and suggested providing more examples and discussion on multi-category interpretation challenges. To be more conclusive regarding the assertion ''Does that demonstrate that when there is a multi-category in the image, the model cannot be interpreted well in the wavelet domain'', we are gathering more examples. Localization benchmarks should also give us information on the behavior in multi-category settings. We try our best to get the results and will update the rebuttal.

Questions

1. adopting wavelet in the feature layer;

This is an interesting question and was actually planned for future work. To the best of our knowledge, no work has adopted wavelets in the feature layers. Like [A,B] and contrary to the statement of the reviewer, the wavelet decomposition does not prevent us from using the intermediate layers of the model to compute the explanations. In principle by applying wavelet decomposition to intermediate feature maps, we could obtain a "multiscale interpretation" of feature importance.

2. wavelet-based explanation method for interpreting Transformer;

If the reviewer refers to the interpretation of the attention mechanism using wavelets, it is an interesting suggestion but beyond the scope of this work. If it is meant applying WAM to Transformer-based architectures, then we kindly refer the reviewer to the supplementary materials B2 were we apply the WAM to a wide variety of topologies (ViT, ConvNext, 3D Former). Results remain the same no matter the choice of the classification model.

3. wavelet-based explanation method for other task models like object detectors and VLMs

These two comments are a very good suggestion and we would like to thank ther reviewer for these remarks. We remained focused on the classification task but in principle, our method could be expanded to object detectors. The way to follow would be to expand ODAM to the wavelet domain. Regarding VLMs, since our method unifies the domain for feature attribution it makes sense to look for interpreting multimodal models. However VLMs handle text data, and this modality is unsuitable for the wavelet transform, so attribution in a suitable latent space for text data shoud be explored. We are currently applying WAM to object detectors and will update the rebuttal if we get results.

We first would like to thank reviewer KCeN for reviewing our manuscript and for the comments on our work. The points raised by the reviewers will help us improve the quality of our work.

Question Pg 4, line 209, left: [...] Please expound / I can see how wavelets is another way to attribute to the input domain, so that attributions now incorporate a sense of scale. [...] can you provide any argument as to why wavelets are appropriate terms in which to attribute? What about super-pixels? What about Fourier transforms and attributing in k-space? Is this method more appropriate than those, theoretically or experimentally?

Our formulation "varying pixel values provide no information to wht is changing on the image" is unclear and we modified it for "moving from one pixel to the next is only a shift in the spatial domain and does not capture relationships between scales or frequencies". This sentence explains why in our view, the pixel domain is insufficient to interpret the decision of a model. The wavelet domain captures both the spatial component (as done by pixels, or superpixels) and the spectral component, as done by the Fourier transform. Either one of these approaches is unsuitable for attribution (see [1,2] for Fourier), as for attribution a spatial or temporal dimension is required and a spectral dimension desirable to assess what the model sees at a given location.

Question What wavelet functions were chosen for each experiment? What was chosen? Why were they chosen? [...] perhaps they should be presented more prominently. / How does the faithfulness score and the quality of explanations vary with the choice of wavelet function and choice of $\Lambda$ ? Is it sensitive to these choices?

The quality and faithfulness of explanations does not change with the choice of the mother wavelet. the quality and faithfulness of the explanation stem from the multi-scale decomposition, which is a property of the $\Lambda$, irrelevant of the choice of the mother wavelet. The choice of the mother wavelet determines how the input signal is decomposed, influencing whether finer details or broader structures are emphasized in the wavelet coefficients By default, we considered the Haar wavelet, due to its semantic properties but also because it is fast to compute.

Additional experiments verified that the choice of the wavelet does not change the quantitative results. We evaluated $WAM_{IG}$ on a ResNet 50 model using Daubechies and Bior wavelets. We updated the supplementary materials in to state more clearly our choice of wavelet and discuss what wavelets can be chosen.

On the other hand, the choice of $\Lambda$, remained fixed and we did not explore beyond the dyadic transform because the multi-scale property of our decomposition fundamentally relies on its dyadic structure. Using non-dyadic decompositions could disrupt the natural hierarchy of scales, leading to a loss of spatial localization and a less structured frequency representation.

Question Your paper claims WAM is a unifying perspective of feature attribution. [...]. However, the paper seems to provide a method that is adaptable to multiple input domains. Perhaps some more up-front clarity/renaming would clarify this fact and prevent confusion.

We acknowlege that we remained elusive on what we aimed at unifying, a concern shared by R3. We will edit our manuscript to explicitely state what we aim to unify in this work.

Question A major component of IG is completeness. [...] Does WAM_IG satisfy completeness?

For completeness, only showing that inverse wavelet transform keeps things differentiable and that the initial and final points are same is enough (see [3], Prop 1) to show that $WAM_{IG}$ satisfies completeness.

Formally, IG satisfies completeness if $F$ is differentiable almost everywhere and

$$\sum_{i=1}^n IntegratedGrad_i(x) = F(x) - F(x_0)$$

Where $x_0$ is a baseline black image such that $F(x_0)\approx 0$. To ensure that $WAM_{IG}$ satisfies the condition for completeness, we have to ensure that (1) $F(W^{-1}(z))$ is differentiable almost everywhere and that (2) we can set $z_0$ such that such that $W^{-1}(z_0)$ is a black image. (1) depends on the choice of the mother wavelet $\psi$ (we need the mother wavelet to be smooth) and (2) is obtained by setting the wavelet coefficients to 0, so $WAM_{IG}$ satisfies completeness if $\psi$ is smooth, e.g. with Daubechies but not with Haar wavelets.

• [1] Yin et al (2019). A fourier perspective on model robustness in computer vision. NeurIPS
• [2] Chen et al (2022). Rethinking and improving robustness of convolutional neural networks: a shapley value-based approach in frequency domain. NeurIPS
• [3] Sundararajan et al (2017) Axiomatic attribution for deep networks. ICML