Understanding Parameter Saliency via Extreme Value Theory

We appreciate your careful considerations on improving the work!

Questions and Answers

Q.1 Do we really need a more complicated distribution for this purpose now?

A.1 We are actually further relaxing the assumption of a normal distribution. Although EVT may seem to require more assumptions, it is known that extreme values of many standard distributions follow this distribution, and the normal distribution is no exception. It is known that the maximum value from the normal distribution converges to the Gumbel distribution. It also has been demonstrated that EVT can handle various distributions, including exponential and uniform distributions. Therefore, there is no implicit need to assume a normal distribution, making it applicable to a broader range of distributions.

Q.2 What's the limitation in the previous solution(Levin) and how this solution can solve that?

A.2 As stated in Section 4.1, assuming a normal distribution itself can be considered a limitation. This is particularly true when dealing with heavy-tailed distributions, where values like mean and variance are significantly influenced by outliers, making them unreliable as proper statistical measures. In the existing method (Levin et al.), the use of z-score transformation may be strongly affected by this, potentially being unable to achieve accurate comparisons. Moreover, comparing values calculated from statistics of distributions that are not necessarily the same may inherit the characteristics of each distribution. For instance, by examining the tables in Section 5.3, it becomes evident that the existing method struggles to select filters from the early convolutional layers. This is attributed to the high magnitude of gradients and increased variance in the early layers. In contrast, the POT method transforms the discussion into a probabilistic context, enabling a unified comparison irrespective of the underlying distribution.

Q.3 I might miss something in the paper, proposition 1 and appendix B also are based on normal distribution, can I ask why a simple normal distribution can be representative enough in this problem raised?

A.3 Proposition 1 and Appendix B pertain to the methodology of Levin et al. In their approach, they employ z-score transformation as depicted in Equation (2). While z-score transformation implicitly assumes a normal distribution, Proposition 1 asserts that under such an assumption, the ranking score function can be viewed as a probability. By introducing a probabilistic perspective in this manner, a new method utilizes extreme value theory to enable statistically meaningful comparisons.

Others

• Thank you for pointing out the inconvenience of the legend and colors in the figure. We will fix it.
• For the suggestion on the related work, parameter saliency is a new concept and related work is really scarce. Also, POT is rarely used in the context of machine learning. It is rather used in fields like hydrology, finance, and environmental science. So we consider XAI related works better fit the section.

We appreciate your thoughtful feedback, which has been invaluable in improving the quality of our work.

Q.1 How does the proposed POT algorithm work? A flowchart may be used to help the discussion. ( how to link Theorem 1 in Eqs. 3 and 4 with the POT method?)

A.1 Thank you for your suggestion. We have added a flowchart illustrating the operation of the POT algorithm in Figure 6 of the appendix. If you have any uncertainties, please feel free to inquire with us again.

Regarding the relationship between Theorem 1 and the POT method, Theorem 1 means that the more you move towards the tail of the probability distribution, the better it can be approximated by the Generalized Pareto Distribution (GPD). The POT method is a technique for estimating the parameters of this GPD, utilizing extreme values exceeding a sufficiently large threshold for maximum likelihood estimation.

Q.2 More comparison experiments need to be conducted but the current version is weak in terms of experimental work.

A.2 We will add some comparison experiments within the next few days, we would appreciate it if you could wait until then.

Q.3 How will the parameters in the proposed POT system affect the system performance?

A.3 One critical parameter to consider for POT is the threshold. Generally, if the threshold is set too high, the amount of data available for estimating GPD parameters becomes limited, resulting in increased variance. On the other hand, if the threshold is set too low, the approximation accuracy of GPD tends to deteriorate.

In terms of the POT method, it is worth noting that since the algorithm generates rankings for filters that exceed the threshold, the number of filters included in the ranking is influenced by the choice of the threshold. This is the largest difference between POT method and Levin et al.'s method because the latter one always gives us the ranking of the size of the total filter number.

We did an experiment on ImageNet dataset by setting the threshold to 95-th percentile of gradient data, but it did not show much difference. This can be because the ImageNet dataset is very large and the threshold has a little impact on maximum likelihood estimation of POT parameters.

Q.4 The outcomes of the experiments are not significant, compared against those of the other state of the arts.

A.4 We claim that the primary objective is to comprehend the existing research (Levin et al.) from the perspective of statistical anomaly as the title suggests. Although we compare several approaches in the experiments, parameter saliency is a concept proposed recently and there is a "knowledge gap in terms of understanding why parameter saliency ranking can find the filters inducing misidentification" as we stated in the introduction. Therefore, we do not consider it as weakness.

Thank you sincerely for your thoughtful and constructive feedback on our paper.

Questions and Answers

Q.1 Is the finetuning and validation being done on the same dataset ?

A.1 In general, the dataset used for finetuning is distinct from the validation dataset. When using a pre-trained model provided by deep learning frameworks, there is no access to the actual validation set during training. Also ImageNet officially offers two datasets, “training set” and “validation set”, where people commonly split “training set” for training and validation. In our work and existing work(Levin et al.), “validation set” is used for calculating the statistics and performing one-step finetuning.

Q.2 The interpretations in Figure 4 are not clear, especially the conv5_x curve and its relation to the proposed approach.

A.2 There is no specific relationship between conv5_x and POT. As described in section 5.1.3, Kirichenko et al. reported good performance when finetuning only the final layer. Additionally, in Tables 1 and 2 of Section 5.3, it can be observed that the approach of Levin et al. is heavily biased towards the latter convolutional layers. The conv5_x method is developed based on these observations. The conv5_x curve in Figure 4 demonstrates that the model performance improves by only finetuning the convolutional filters in the latter layers in descending order of gradient magnitude. This raises the question of whether it is necessary to modify feature extractor in the earlier layers when dealing with scenes requiring different features due to domain shift.

Q.3 I think it will be more intuitive to think of %improvement in downstream performance rather than %corrected samples.

A.3 Our experimental metric aligns with the one used in the existing study (Levin et al). The objective of Levin et al.'s research is to identify which part of the model parameters is responsible for misclassifications. Investigating %corrected samples is effective because if correcting a limited number of filters through one-step finetuning leads to accurate classification, it validates the saliency of those filters.

Q.4 How do you differentiate between label errors and misclassification ?

A.4 Our study is predicated on the occurrence of misclassifications. The primary objective is to comprehend the existing research (Levin et al.) from the perspective of statistical anomaly. The experiments and problem settings are conducted based on this research. Consequently, the analysis is carried out in an ideal scenario, assuming there are no factors such as label errors.

Q.5 I am not too convinced with the one-step finetuning process apart from being "easy" to do.

A.5 This is a good point. We again note that one-step finetuning employs the same experimental methodology as the existing study (Levin et al.) and is utilized for evaluation. We would like to examine changing the salient parameters can positively affect the classification result. Also, since the statistical measures, such as the average magnitude of gradients, would change with model parameters, conducting multiple steps of finetuning would be a more complex way of evaluation. Thus, one-step finetuning is used.

Q.6. the anomalous behavior of filters causing misclassification can also be considered a rare event" - Can the authors give more insight into when would this hold ? Perhaps as a function of class difficulty, model size, dataset size.

A.6 In the context of gradients, the simplest explanation can be derived. As illustrated in Figure 1, there is a notably high frequency of instances where the magnitude of the gradient is nearly zero. This phenomenon occurs when the deep learning model correctly classifies the input. In other words, as the model's performance improves, instances where the gradient magnitude is significantly larger than zero become increasingly rare. Therefore, it is hypothesized that this becomes more achievable with larger dataset sizes and model sizes.

Others

• For experiments on domain adaptation datasets, we will add them within the next few days, so we would appreciate it if you could wait until then.
• Thank you for writing suggestions. We addressed them and updated our draft.