AttEXplore: Attribution for Explanation with model parameters eXploration

Weaknesses:

1. Thank you for the thoughtful consideration of our work. It is necessary to generalize our method to models for different tasks beyond image classification. Working on the interpretability of NLP models is a good direction. For example, verifying the interpretability of medical reports generated by applying NLP technology. At present, our work is aimed at attribution interpretation of image related tasks, which is consistent with the current SOTA research works including AGI [1], BIG [2], GIG [3], and SG [4].

2. Given that our method satisfies the sensitivity and implementation invariance axioms in IG [5], this ensures a one-to-one correspondence between the model's input and output during attribution, guaranteeing the accuracy of the attribution process. We have provided detailed proofs of our method's adherence to these axioms in the appendix.

3. We appreciate this suggestion. However, attribution methods are model agnostic interpretability methods, which is different from the interpretable-by-design approaches for white-box transformers. While Vision Transformers (ViTs) represent a typical application of the Transformer model in computer vision tasks, we have additionally included the experiment result of our method on the ViT model. We anticipate the result in the global comments Table *4 may be helpful to further extend the comparison in the future to evaluate its interpretability performance in this domain. We will include the discussion with this specific white-box transformer [6] work in our revision.

Reference:

[1] Pan, D., Li, X., & Zhu, D. (2021, August). Explaining deep neural network models with adversarial gradient integration. In Thirtieth International Joint Conference on Artificial Intelligence (IJCAI).

[2] Wang, Z., Fredrikson, M., & Datta, A. (2022). Robust models are more interpretable because attributions look normal. Proceedings of the 39th International Conference on Machine Learning.

[3] Kapishnikov, A., Venugopalan, S., Avci, B., Wedin, B., Terry, M., & Bolukbasi, T. (2021). Guided integrated gradients: An adaptive path method for removing noise. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 5050-5058).

[4] Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017). Smoothgrad: removing noise by adding noise. the 34th International Conference on Machine Learning.

[5] Sundararajan, M., Taly, A., & Yan, Q. (2017, July). Axiomatic attribution for deep networks. In International conference on machine learning (pp. 3319-3328). PMLR.

[6] Yu, Y., Chu, T., Tong, S., Wu, Z., Pai, D., Buchanan, S. and Ma, Y., 2023. Emergence of segmentation with minimalistic white-box transformers. arXiv preprint arXiv:2308.16271.

Evaluation Shortcomings:

Thanks for the reviewer's comments. Our method satisfies the Completeness axiom, as proven in Section 3 of IG [3], serving as a sanity check. We have also conducted INFD score tests [4] in the global comments in Table *2, and the results demonstrate that our method exhibits superior robustness and fidelity.

Concerning LIME [2], LRP [5], and SHAP [6], we would like to note that these methods have different requirements compared to current state-of-the-art attribution methods. Firstly, LIME attempts to perturb inputs to identify the inputs most responsible for a prediction in the neighborhood. In addition, LIME’s interpretable behavior requires cluster segmentation of images, so its interpretation model is not point-to-point. And if it performs point-to-point calculations, the computational complexity of LIME will be extremely high. LRP relies on model structure and is challenging to implement. DeepLIFT is a more general case of LRP, and our method surpasses DeepLIFT. SHAP, when applied to samples with high dimensions, uses approximate results for inference and involves a significant amount of interaction operations, resulting in extremely high computational complexity.

Secondly, IG [3] extensively discusses that these methods do not necessarily satisfy the sensitivity and implementation invariance axioms. For example, LRP uses discrete gradients to compute feature importance, which does not satisfy the sensitivity axiom and may lead to instances where attributions change, but the attribution result is 0, rendering interpretability completely ineffective.

Lastly, current state-of-the-art methods including AGI [7], BIG [8] and GIG [9] don’t include LIME, LRP, and SHAP as the comparison in their experiments. It is noteworthy that some of the methods are noise reduction-based methods which were published much earlier. In our baseline, GIG represents the state-of-the-art noise reduction-based method as proposed in 2021. We have achieved significant improvement over the GIG method.

Regarding the subset selection for time complexity comparison, we selected the five methods that are closest in performance to our method for efficiency comparison. Other methods, such as SaliencyMap, DeepLIFT and EG have relatively poor attribution accuracy in the baseline, making them unnecessary for comparison. Although Fast-IG achieves faster attribution speeds, its accuracy is unfortunately the lowest.

Reference:

[1] Shrikumar, A., Greenside, P., & Kundaje, A. (2017, July). Learning important features through propagating activation differences. In International conference on machine learning (pp. 3145-3153). PMLR.

[2] Ribeiro, M. T., Singh, S., & Guestrin, C. (2016, August). " Why should i trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1135-1144).

[3] Sundararajan, M., Taly, A., & Yan, Q. (2017, July). Axiomatic attribution for deep networks. In International conference on machine learning (pp. 3319-3328). PMLR.

[4] Yeh, C. K., Hsieh, C. Y., Suggala, A., Inouye, D. I., & Ravikumar, P. K. (2019). On the (in) fidelity and sensitivity of explanations. Advances in Neural Information Processing Systems, 32. [5] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K. R., & Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7), e0130140.

[6] Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. Advances in neural information processing systems, 30.

[7] Pan, D., Li, X., & Zhu, D. (2021, August). Explaining deep neural network models with adversarial gradient integration. In Thirtieth International Joint Conference on Artificial Intelligence (IJCAI).

[8] Wang, Z., Fredrikson, M., & Datta, A. (2022). Robust models are more interpretable because attributions look normal. Proceedings of the 39th International Conference on Machine Learning.

[9] Kapishnikov, A., Venugopalan, S., Avci, B., Wedin, B., Terry, M., & Bolukbasi, T. (2021). Guided integrated gradients: An adaptive path method for removing noise. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 5050-5058).

Clarity:

Thanks for the constructive comment. We will rewrite the relevant sections of abstract and introduction to facilitate a more seamless understanding for readers. Additionally, we will improve Figure 1, enhancing image clarity and the representation of key elements to provide a clearer understanding of our method. We will also refine the structure of the appendix and provide more precise and specific references to its content within the main text for improved citation.

Regarding the statement "DeepLIFT and LIME generate explanations for generic models which limits the understanding of the model’s global behaviors," our intention is to convey that DeepLIFT [1] and LIME [2], as local interpretability methods, primarily focus on explaining the local neighborhood behaviors of models. These methods typically employ simpler, interpretable models to approximate the behavior of complex models near specific data instances or attempt to perturb inputs to identify the inputs most responsible for a prediction in the neighborhood. Therefore, these two methods do not satisfy the sensitivity and implementation invariance axioms proposed by IG [3], resulting in limited global interpretability of model behaviors. Global interpretability refers to understanding the entire behavior of the model, not just explanations for specific instances or local regions. Global interpretability focuses on properties such as features, weights, structure, and behavior patterns of the model across the entire input space.

In Equation (4), the variable $i$ represents the number of frequency domain explorations, i.e., $i=1,2,...,N$. Ultimately, we generate $N$ adversarial samples through frequency domain exploration. We will review the main context to avoid similar ambiguities.

Thank you for your constructive comments. GradCAM [1] needs to select a layer in the middle (the dimension is lower than the original image dimension and has multiple channels). Feature channels are combined based on the gradient sum of each channel. Because of the low dimensionality, pixel-level attribution cannot be achieved. In addition, since it only uses the gradient of a single state, similar to Saliency Map [2], it does not satisfy Sensitivity. Although we believe that GradCAM is not an attribution algorithm in the strict sense (does not satisfy the definition and axioms of attribution) and it cannot be on the same track as our method, we are happy to provide the comparison results of GradCAM. While we believe this is not necessarily fair, our method still perform better than GradCAM. Because our method is better than GradCAM in both insertion score and deletion score.

In order to prove the effect of our method on other models, we also conducted additional experiments on the currently widely used Vision Transformer (ViT) models. The experimental results can be seen in Table 4 in the global comment. Our method can still achieve the best results on the ViT models.

We will update all details of this experiment to the anonymous link of the code.

We apologize for our honest fault in the rebuttal reply in Official reply to Reviewer r7kS (Part 2), which is a typo for 'Other methods, such as SaliencyMap, involve extensive computations, making them slower'. In fact, as you point out, Saliency Map is more efficient. We would like to clarify that Other methods, such as SaliencyMap, DeepLIFT [3] and EG [4] have relatively poor attribution accuracy in the baseline, making them unnecessary for comparison. We will correct this typo in the original reply.

We have listed the speed analysis table of the algorithm in Section 5.3 of the article. It can be seen that the efficiency of our algorithm has achieved the optimal effect in the IG-based attribution method. On the other hand, we believe that the faithfulness of attribution is more important than the efficiency of the interpretability.

[1] Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision (pp. 618-626).

[2] Karen Simonyan, Andrea Vedaldi, Andrew Zisserman: Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. ICLR (Workshop Poster) 2014

[3] Shrikumar, A., Greenside, P., & Kundaje, A. (2017, July). Learning important features through propagating activation differences. In International conference on machine learning (pp. 3145-3153). PMLR.

[4] Erion, G., Janizek, J. D., Sturmfels, P., Lundberg, S. M., & Lee, S. I. (2021). Improving performance of deep learning models with axiomatic attribution priors and expected gradients. Nature machine intelligence, 3(7), 620-631.

Thanks to the reviewer for the response. We hope that the following detailed discussion will clear up your confusion:

First of all, we would like to correct our point of view. In fact, DeepLIFT [1] does not all satisfy the two axioms proposed by IG [2]. Although DeepLIFT does not require a simpler interpretable model or perturbs inputs, and satisfies the sensitivity axiom, DeepLIFT breaks Implementation Invariance axiom. The corresponding original text in IG is at the beginning of Section 2: We now discuss two axioms (desirable characteristics) for attribution methods. We find that other feature attribution methods in literature break at least one of the two axioms. These methods include DeepLift ( Shrikumar et al., 2016; 2017), Layer-wise relevance propagation (LRP) (Binder et al., 2016), Deconvolutional networks (Zeiler & Fergus, 2014), and Guided back-propagation (Springenberg et al., 2014) .

A more direct explanation can be found in the second paragraph of Section 2.2: We now discuss intuition for why DeepLift and LRP break Implementation Invariance.

Our attribution method satisfies the two axioms of sensitivity and implementation invariance, which enables one-to-one correspondence between model input and output to obtain more accurate attribution results.

Secondly, gradient-based and perturbation-based methods are currently two important directions in interpretability methods. LIME [3] or other perturbation-based methods obtain the contribution of the model input to the output through perturbation. But this is not consistent in principle with our gradient-based attribution method, where we leverage the backward pass of a neural network to assess the influence and relevance of an input feature on the model decision. In addition, as you said, LIME requires a large amount of calculation, which is one of its disadvantages.

[1] Shrikumar, A., Greenside, P., & Kundaje, A. (2017, July). Learning important features through propagating activation differences. In International conference on machine learning (pp. 3145-3153). PMLR.

[2] Sundararajan, M., Taly, A., & Yan, Q. (2017, July). Axiomatic attribution for deep networks. In International conference on machine learning (pp. 3319-3328). PMLR.

[3] Ribeiro, M. T., Singh, S., & Guestrin, C. (2016, August). " Why should i trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1135-1144).

Weaknesses:

1. Thanks for pointing out the valuable comment for attribution method research. Actually, exploring interpretability in the context of NLP models is undoubtedly a valuable direction, especially in domains such as medical report generation where interpretability is crucial. Currently, our work focuses on attribution interpretation for images, aligning with the current state-of-the-art approaches like AGI [1], BIG [2], GIG [3], SG [4], and other baselines. We would like to provide a discussion of potential work following the comment for NLP works.
2. We appreciate the reviewer's constructive suggestions. We have conducted the additional experiment on the model of Vision Transformers (ViTs). ViTs represent an excellent vision model based on the Transformer architecture. Different from traditional DNN models, ViTs process images as sequences of patches, leading to a patch-like interpretation of features [5]. The experimental results on ViTs, included in the global comments Table *4, still demonstrate that our method achieves the best performance.

Questions:

We are afraid the answer is negative for the trend of model performance under different $N$, even with a larger perturbation rate.

On one hand, the ablation experiments in Tables 3, 4 and 5, and the appendix file, indicate that increasing the number of generated features or perturbation rates, among other hyperparameters, can enhance attribution accuracy to some extent. This is rational as the increase in the number of generated features ($N$) implies the exploration of more frequency domain information, while an increase in $\epsilon$ signifies the generation of higher-quality adversarial samples. Consequently, the performance at a perturbation rate of 48 may surpass the result with a perturbation rate of 16.

On the other hand, the impact of altering hyperparameters on performance, as observed in ablation experiments, is relatively small. This suggests that the improvement of our method does not heavily rely on the specific choice of hyperparameters. However, when the perturbation rate is set to a larger value such as 48, the trend in model performance across different $N$ values does not become more clear. This might be attributed to the fact that a larger perturbation rate represents a larger search space. Although the number of explored samples increases, it doesn’t mean that all these samples are necessarily effective for the attribution result. We have validated this observation in the additional experiments provided in the global comments Table *5.

Reference:

[1] Pan, D., Li, X., & Zhu, D. (2021, August). Explaining deep neural network models with adversarial gradient integration. In Thirtieth International Joint Conference on Artificial Intelligence (IJCAI).

[2] Wang, Z., Fredrikson, M., & Datta, A. (2022). Robust models are more interpretable because attributions look normal. Proceedings of the 39th International Conference on Machine Learning.

[3] Kapishnikov, A., Venugopalan, S., Avci, B., Wedin, B., Terry, M., & Bolukbasi, T. (2021). Guided integrated gradients: An adaptive path method for removing noise. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 5050-5058).

[4] Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017). Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825.

[5] Ma, W., Li, Y., Jia, X., & Xu, W. (2023). Transferable Adversarial Attack for Both Vision Transformers and Convolutional Networks via Momentum Integrated Gradients. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4630-4639).

Evaluations:

Regarding Tables 3, 4 and 5, these results are for the ablation study, which is to evaluate the impact of different hyperparameters. The conclusion is that, by increasing the number of generated features or perturbation rates, the attribution accuracy may be slightly increased. This is also apparent since an increase in $N$ implies the exploration of more frequency domain information, and an increase in $\epsilon$ signifies higher-quality adversarial samples. However, we also acknowledge that the improvement is not substantial, indicating that our method's effectiveness does not heavily rely on hyperparameter selection, showcasing higher generalization and fidelity. We also provide the performance results utilizing different transferable attack methods in the appendix, which proves the effectiveness of our method.

Concerning the $\epsilon$ value, the value of 8 in ablation experiments is not large as we have followed the literature method from [2]. Typically, only when $\epsilon<=16$, we consider the samples can reliably cross more general decision boundaries for a satisfactory attack outcome. We have now conducted more additional experiments to demonstrate that our method doesn’t need particular hyperparameter selection.

For the INFD score and sanity check, our method is faithful to the underlying model, as demonstrated in Section 3 of IG [3], which has proven the Completeness (Sensitivity) axiom as a sanity check. We did not utilize Faithfulness [4] as a fidelity metric because its calculation involves random selection of a subset, and the subset's size and randomness in selection will introduce significant bias to the results. To mitigate this bias, a more extensive range of sampling is required, leading to substantial computational costs. Thus, we have been mindful when employing such a stochastic evaluation metric, and we would like to note that for AGI, BIG, GIG and similar methods, they also do not adopt this metric in their experiment.

In the global official comment, we provide INFD score results [5] in Table *2, and conduct ablation experiments with lower $\epsilon$ values in Table *3, demonstrating that our method exhibits the best performance of robustness and fidelity.

Reference:

[1] Zhu, Z., Chen, H., Zhang, J., Wang, X., Jin, Z., Lu, Q., ... & Choo, K. K. R. (2023, October). Improving Adversarial Transferability via Frequency-based Stationary Point Search. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (pp. 3626-3635).

[2] Zhang, J., Wu, W., Huang, J. T., Huang, Y., Wang, W., Su, Y., & Lyu, M. R. (2022). Improving adversarial transferability via neuron attribution-based attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14993-15002).

[3] Sundararajan, M., Taly, A., & Yan, Q. (2017, July). Axiomatic attribution for deep networks. In International conference on machine learning (pp. 3319-3328). PMLR.

[4] Bhatt, U., Weller, A., & Moura, J. M. (2021, January). Evaluating and aggregating feature-based model explanations. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence (pp. 3016-3022).

[5] Yeh, C. K., Hsieh, C. Y., Suggala, A., Inouye, D. I., & Ravikumar, P. K. (2019). On the (in) fidelity and sensitivity of explanations. Advances in Neural Information Processing Systems, 32.

Motivation:

We appreciate the comments about the motivation and are sorry about the confusion.

We would like to clarify a misunderstanding about the comparison with BIG and AGI (which are the SOTA methods), by highlighting the novel contribution which is also noted by other reviewers. The technical contribution is actually to leverage the insight from transferable adversarial attacks.

The motivation is to enhance the attribution performance by considering different decision boundaries with the insights from transferable adversarial attacks. We consider the features to be crucial for the model output if the altered sample will cross the decision boundary. The summarization of the contributions of these features is considered as the attribution results. However, the training process from literature methods is limited to its capability of finding the accurate decision boundary (most variations in the decision boundary are OOD samples). How to effectively identify and leverage the decision boundary is challenging.

Thus, inspired by the exploration of decision boundaries in transferable attacks, we observe the consistency of such adversarial sample generation with the samples for attribution. The reason is that a transferable adversarial sample on a given model also needs multiple operations of exploring the decision boundaries for a robust result. In this way, we consider the obtained sample will be able to cross more decision boundaries without the parameters from the target model, which means a more accurate attribution result. More discussion is available in Section 4.1, where we provide the reasons about the possibility of utilizing the samples to explore the decision boundary during the training process.

Moreover, we argue that the performance of current attribution methods is generally influenced by the choice of different baselines. When the underlying task is complex, selecting an appropriate baseline is often difficult and ad-hoc. Therefore, for the decision boundaries of specific models corresponding to different underlying tasks, the local explanation of IG may not be optimal. For BIG, it uses a linear attribution path based on adversarial samples, but it cannot effectively solve this problem because linear attribution paths are unstable when crossing the decision boundaries of nonlinear complex models. For AGI, although it incorporates adversarial samples and nonlinear attribution paths other than the subjective selection of baselines, it aims at targeted adversarial attack, which may cover some unnecessary decision boundaries before successfully exploring the decision boundary corresponding to the target label, potentially leading to overfitting issues.

In conclusion, to obtain a decision boundary that allows samples to cross without overfitting concern, we consider directly generating multiple decision boundaries with slight offsets through parameter adjustment is not practical and effective. Therefore, we conduct input exploration with transferable adversarial attack to simulate the process of sample generation by crossing the decision boundaries, thereby obtaining more accurate attribution.

Generalization of Frequency-based Methods?

Please see the analytical results provided in the global official comment in Table *1-1 and *1-2, which shows the result of attack success rate with different transferable adversarial attack methods. The generated adversarial samples are tested for the attack success rate on other victim models, in which a higher attack success rate represents a higher potential for crossing other general decision boundaries [1]. We extend our discussion of the motivation to leverage the insights from transferable attacks for decision boundaries exploration to further enhance the attribution method performance. In practice, it means that, if the adversarial samples can effectively cross various decision boundaries, it implies a higher transferability for transferable attacks and, for attribution, a better capability to obtain more accurate identification of critical features. Additionally, we emphasize that our findings extend beyond frequency domain transformations. We have tested the impact of different input transformations for interpretability.

Thanks to the reviewer for the response.

Clearer Motivation clarification:

It is obvious that if an input feature can cross the decision boundary of the model and change the decision-making of the model, then this input feature can be attributed as a high-contribution feature, that is, an important feature. However, we need to consider the impact of inaccurate decision boundaries during model training, because usually, the confidence of the training data is high and far away from the decision boundaries. That is to say, those samples that are very close to the decision boundary are likely to be OOD samples and are prone to overfitting.

We believe that since the transferable attacks can more generally cross the decision boundary of the target model, the decision boundary obtained by the transferable attacks is likely to be less overfitting than the original decision boundary, in other words, more accurate. Therefore, it is clear whether we can use the characteristics of transferable attacks as our motivation for attribution. Because the relationship between attribution and decision boundaries is very obvious. We also illustrate the equivalence between decision boundaries and input exploration in the paper.

Clearer Generalization clarification:

Regarding your original comment, 'I am pretty worried the authors explain this conclusion as altering some features in the frequency domain helps to create examples that transfer better, especially for methods just proposed in this paper (Eq. 4 - 5). Have you verified your adversarial examples actually transfer better? Can you provide some analytical results convincing me this helps better explore more decision boundaries?'

Our purpose in adding FDE experiments is to implement the proposed requirements, which also shows that our method can achieve better transferable attack results (meaning more decision boundaries are crossed). We do not attempt to prove that the transferability of FDE is better, since our goal is focusing on attribution. The cited baselines are from recent years to illustrate that there is indeed an impetus to crossing more general baselines. Similarly, comparing all transferable attack methods is not realistic. For us, we have selected the most representative and SOTA works to prove the effectiveness of FDE. Our goal is not to create a better transferable attack but to illustrate the effectiveness of frequency exploration for attribution. Always, there will be better ways to move the attacks, but that's not the focus of this paper, not either the job for attribution.

For the comparison process, we used the same open-source code and the same experimental design as the baselines in most recent years. And we have added all the details in the anonymous code.

Clearer clarification of Evaluations:

Our experiments show that the larger $epsilon$ does have an impact on the results, and it shows an increasing trend. Together with Table 5 (in our submission), Table *5, and the Table *3, the impact of $epsilon$ cannot be ignored. From the literature, the general setting for $epsilon$ is 16, which is actually larger than 8. In addition, our attribution process also requires multiple gradient ascents, so even a very low $epsilon$ will have an effect after multiple gradient ascents. This is not counter-intuitive.