On the Generalization of Gradient-based Neural Network Interpretations

Update: I would like to thank the authors for their response, and I have read all the reviews and the corresponding responses. I decide to keep my original score as my main concerns remain. Specifically, I remain unconvinced about the significance of studying the generalization of interpretation methods. Consider the reference classifier in your revised version, namely $f^*(x)=\mathbb{E}\_{f\sim\mathrm{Unif}(F^*)}f(x)$, the question arises: is $\mathbb{E}\_{f\sim\mathrm{Unif}(F^*)}||I(f,x)-I(f^*,x)||_2$ small? As mentioned in my previous question 2, if the optimal $f$ set can have large variance/standard deviation, why does it matter if the saliency maps are sensitive to the training data?

This paper investigates the generalization of gradient-based interpretation methods in deep learning. The authors demonstrate the significant influence of the sample size on the interpretation, such as saliency maps, in deep neural networks. Then they derive two generalization bounds for common gradient-based interpretation techniques by using the analysis presented in Bartlett et al. (2017). Notably, for SmoothGrad, they show that the generalization error of interpretation has a linear decrease with the standard deviation of the SmoothGrad noise. The paper complements these findings with numerical results demonstrating the impact of training sample size on interpretation outcomes.

The work derives generalization bounds incorporating gradient based interpretations, which yield non-trivial results. It shows that the generalization of interpretations requires more training, and show that it can be improved with spectral normalization.

The paper studies the problem of generalization of gradient-based saliency maps of deep networks. Theoretical bounds are shown and experiments are carried out to validate the results.

We thank the reviewers for their time and feedback on our work. Here, we respond to the common comments raised by Reviewers Yqqc and Cg7V. We provide our responses to the other questions and comments under each review.

1- Motivation of our study of the generalization of interpretation maps

Re: In the deep learning literature, there are several widely-known methods for generating interpretation maps for neural net classifiers, including SimpleGrad, Integrated Gradients, DeepLIFT, SmoothGrad, Sparisified SmoothGrad, etc. Also, the application of these methods requires the selection of some hyperparameters, e.g. the standard deviation parameter in SmoothGrad. Therefore, an essential question is how and based on what factors a practitioner should select an interpretation map and its hyperparameters from these established methods.

We argue that the “sample complexity” of estimating an interpretation map should be a factor in choosing the interpretation method and its hyperparameters. To understand the importance of sample complexity, we consider the following scenario from our CIFAR10 experiments in Section 6 where the expected norm difference between normalized SimpleGrad maps generated for two neural nets $f_{S_1},f_{S_2}$ learned under two independent training sets is $0.242$, while the same expected norm difference between normalized SmoothGrad maps (with $\sigma = 1$) is $0.019$. Then, a natural question is whether a practitioner should interpret and act based on the SimpleGrad map that would have been significantly different if the neural net had been trained using an independent training set, while the SmoothGrad map would behave with higher stability to the randomness of training data.

In this work, our goal is to bound the generalization error of the interpretation maps as a variance-based quantity with respect to the randomness in the training data. Specifically, our results show that the Simple-grad and Integrated-gradients maps could be highly affected by the randomness of training data, indicating that they may suffer from a high variance under a limited-size training set. On the other hand, we show SmoothGrad maps with sufficiently large variance will have a bounded generalization error, which suggests SmoothGard is a more stable and generalizable map with respect to the randomness of the training process.

To summarize our response, the importance of the generalization study for interpretation maps reduces to one’s answer to the following question: Should the variance of interpretation maps (with respect to the randomness of training data) be a factor for selecting an interpretation scheme and its hyperparameters from the long list of established methods in the literature? We believe the answer to this question is yes and our work aims to better understand the generalizability of gradient-based maps.

2-The role of the population solution $f^*$ in our generalization analysis

Re: Please note that the saliency map $I(f^*,x)$ under the population solution $f^*$ is only a reference interpretation map that we use to quantify the variance of the interpretation map and compare it across training and test data. We note that if the minimizer to the population risk is not unique, we could define $f^*(x) = E_{f\sim \mathrm{Unif}(F^*)}[f(x)]$ as the expectation under the uniform distribution on the set of optimal population solutions $F^*$.

Furthermore, it can be seen that our generalization analysis will similarly hold if one substitutes the reference interpretation map $I(f^*,x)$ with any other reference map as long as that reference map is deterministic concerning the randomness of the training set of the neural net. For example, another practically relevant choice of the reference map could be $E_{S,A}[I(f_{A(S)} , x)]$ as the expectation of the reference map over a random training set $S$ of size $n$ sampled from the data distribution and the stochasticity of a training algorithm $A$ used to find the neural net weights $A(S)$. Then, our interpretation loss for a neural net $f_w$ with weights $w$ in the generalization analysis will be

$\mathcal{L}(f_w,x) = \Vert I(f_w,x) - E_{S,A}[I(f_{A(S)} , x)] \Vert$

In the above, we do not define the interpretation loss with respect to an “optimal neural net $f^*$” and instead measure the closeness of interpretation loss to the expected interpretation map over a random training set of size $n$. Please note that the same generalization results in Theorems 1,2 will hold under the above interpretation loss with a different reference point. We have added Remark 1 in the revised paper including this discussion.

Currently, there are several concerns regarding the significance of studying the generalization of interpretation methods and the extent of the paper's theoretical contribution. The authors should make a revision and improve the motivation for their study.

Reject