Data-centric Prediction Explanation via Kernelized Stein Discrepancy

Thank you very much for your feedback. Below is our response to the concerns and questions you raised.

Moving content from appendix to main paper:

We appreciate the reviewer pointing out the page limitation issue. Indeed, we did not notice the page limit change in the paper call. We have now updated our paper draft by moving some sections into the main paper as suggested by the reviewer. Now the main paper is 10 pages.

Correcting typographical errors:

We greatly appreciate the time and effort you have dedicated to reviewing our paper. In the revised version, we have carefully addressed the suggested improvements.

With regard to the concept of goodness of fit, we would like to clarify that it is not a typo. This term is widely used in statistical analysis to evaluate how well a set of observed values aligns with the expected values predicted by a given model.

Qualitative evaluation:

Including qualitative evaluation by showing image examples is quite common in the sample-based explanation research field [1]. We followed the convention by introducing the corresponding observations to show the difference. We agree that concluding qualitative evaluation alone is somewhat subjective, which motivated us to include further quantitative evaluation that was not included in previous works. However, the qualitative evaluation in our experiments does show weaknesses of existing solutions (e.g., producing the same explanations for different test cases). This observation, at least, shows that the existing approaches produce explanations for a class more than that of the specific test sample, which HD-Explain does not suffer from.

Kernel Evaluation:

We exploited multiple commonly used kernel options and showed they all perform well in practice. We are happy to try more, but we note this approach is proposed to introduce the Kernelized Stein Discrepancy for the post-explanation of models and serve as the baseline for future research. Our proposed method can support various kernels that people may want to choose in practice. Introducing new kernels in the paper won't impact the claim of this work

Predictive confidence:

To ensure the robustness of experimental evaluation and to eliminate the possibility of cherry-picking test samples, in our experiment setup, we always conduct random sampling on the test dataset and evaluate the performance of our model on these subsets. Across all sampled subsets, we indeed didn't find any wrong prediction.

Experimental evaluation validity:

To avoid the generation of invalid samples, and keep the validity of manipulated samples we avoided rotation-vertical flipping. In some cases such as number “6” vertical flipping results in number “9”.

[1] Koh, Pang Wei, and Percy Liang. "Understanding black-box predictions via influence functions." International conference on machine learning. PMLR, 2017.

We acknowledge that exploring additional kernel options enriched this experiment. However, our primary objective here is to highlight the importance of hyper-parameter tuning, providing practitioners with the flexibility to choose the most suitable kernel for their specific needs.

In the literature [1-4], the IMQ kernel function is typically selected due to its good theoretical characteristics and practical effectiveness. As shown in [3], the IMQ kernel, together with the Stein operator provides a KSD that controls weak convergence, and the established guaranteed convergence. We didn't do extra work to validate the image after the horizontal flip. It is due to the observation that model performance can improve significantly with horizontal flip data augmenting during model training, and the model can identify numbers with a horizontal flip. Indeed, humans can identify numbers from a mirror as well. We didn't encounter the 2 and 5 confusion since the flip is only horizontal (versus vertical).

[1] Bénard, Clément, Brian Staber, and Sébastien Da Veiga. "Kernel Stein Discrepancy thinning: a theoretical perspective of pathologies and a practical fix with regularization." Advances in Neural Information Processing Systems 36 (2024).

[2]Chen, Wilson Ye, et al. "Stein points." International Conference on Machine Learning. PMLR, 2018.

[3] Gorham, Jackson, and Lester Mackey. "Measuring sample quality with kernels." International Conference on Machine Learning. PMLR, 2017.

[4] Riabiz, M., Chen, W. Y., Cockayne, J., Swietach, P., Niederer, S. A., Mackey, L., & Oates, C. J. (2022). Optimal thinning of MCMC output. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(4), 1059-1081.

[5] Koh, Pang Wei, and Percy Liang. "Understanding black-box predictions via influence functions." International conference on machine learning. PMLR, 2017.

Prevent undesired results for the image flip on SVHN

We didn't do extra effort to validate the image after the horizontal flip. It is due to the observation that model performance can improve significantly with horizontal flip data augmenting during model training, and the model can identify numbers with a horizontal flip. Indeed, humans can identify numbers from a mirror as well. We didn't encounter the 2 and 5 confusion since the flip is only horizontal (versus vertical).

Thank you very much for your feedback. Below is our response to the concerns and questions you raised.

Moving content from appendix to main paper:

We appreciate the reviewer pointed the page limitation issue out. Indeed, we did not notice the page limit change in the paper call. We have now updated our paper draft by moving some sections into the main paper as suggested by the reviewer. In particular, we moved the HD-Explain* description and one discussion section (the previous Appendix D: Discussion) into the main paper to help reader to capture the intuition of this work.

Potential better performance of Influence Function:

We understand the reviewer's curiosity about seeing if we can push up the influence function's performance by letting it run on the full neural network instead of the last layer. Despite the fact that the Influence Function in this paper was implemented exactly as it was proposed in its original paper, where it is meant to run on the last layer (see section 5.1, page 5 of [5]), we note there is a scalability concern even if we use a smaller neural network. As shown in our paper (Table 1), the complexity of the influence function is bounded by the computation complexity of measuring the Hessian-vector product that needs multiple iterations to approximate. In addition, Resnet-18 is already a small network compared to the models that are used in real applications. Further compression may lead to practical concern even if it shows that the influence function can do better on small models.

Evaluation Setting:

Including qualitative evaluation by showing image examples is quite common in the sample-based explanation research field [5]. We followed the convention by introducing the corresponding observations to show the difference. While we agree that apparently similar images are not necessarily the best explanations, they do have positive correlations. In addition, the qualitative evaluation shows some existing solutions' weaknesses in producing the same explanations for different test cases. This observation, at least, shows that the existing approaches produce explanations for a class more than that of the specific test sample, which HD-Explain does not suffer from.

As the reviewer may be concerned that the qualitative evaluation does not provide sufficient confidence in its explainability, we also included the quantitative evaluation to provide further support. Our quantitative evaluation metrics (hit rate and coverage) come from classic information retrieval literature based on the understanding that the sample-based explanation is essentially a ranking problem (where the explanation algorithm should return a ranked list of training samples as high influencers). Here, we also want to highlight that no agreed or standard quantitative evaluation metrics were proposed in previous work. The previous sample-based explanation often only includes qualitative evaluation demonstrating some side benefit of the proposed algorithms. E.g. data mislabeling debugging.

We agree that the proposed evaluation setting is imperfect in capturing all possible counter-examples, but we note they have served as the best in multiple aspects. E.g. Noise Injection. While the setting looks trivial at first glance, it turns out that existing solutions (the well-known baselines) produce only 10% - 50% coverage -- many test points receive the same set of explanations, disregarding their unique characteristic. This interesting observation suggests the weakness of the existing approaches on such simple cases where it can happen in practice -- find typical medical exam cases with exactly the same symptoms as the new case (in a noise level difference).

Additional Discussion about Section 4.3 Kernel Choices:

In our experimental study, we compare the performance of three well-known kernels: Linear, RBF, and Inverse Multi-Quadric (IMQ), across three image classification datasets. Figure 7 presents the results under two data augmentation scenarios. Overall, the IMQ kernel outperforms the RBF kernel in terms of explanation quality (Hit Rate), with a notable advantage in the Horizontal Flip scenario (Figure 7.c), which appears more challenging than Noise Injection. Additionally, IMQ demonstrates superior performance in Coverage (Figure 7.b). The linear kernel, while performing worse than the other kernels in terms of explanation quality, is significantly more efficient, as illustrated in Figure 7.d. This efficiency makes it particularly useful for large datasets. Compared to the baselines in Figure 5, the Linear kernel enables HD-Explain to excel over other methods in both performance and efficiency.

I thank the authors for their exhaustive responses. I am satisfied with these responses and, while the evaluation setup has its flaws, and that the best setup would be a clinical test, I understand that the current setup is an interesting proxy. I raise my score to 8.

Why data modeling and TRAK are not comparable:

Datamodeling and TRAK are not designed as a model explanation approach but a data attribution estimation approach, where their objective is to obtain the data importance to a prediction task by minimizing the LDS loss directly; they do not need a pre-trained machine learning model in their optimization process. The outcome will be the data attribution list that might contribute to a prediction for a family of models (e.g. particular neural network architecture) rather than any particular pre-trained model. Using data modelling objective as example:

Thank you very much for your feedback. Below is our response to the concerns and questions you raised.

Novelty Recap:

We want to highlight that applying the Kernelized (Conditional) Stein Discrepancy (KSD) to the prediction explanation itself is novel to the problem. As far as we know, HD-Explain is the only approach that adapts KCSD for this purpose. Adaptation of KCSD for explanation purposes is not a trivial effort. In particular, in a goodness-of-fit context, KSD is to quantify the discrepancy between a fitted distribution and the underlying data distribution. The use of KSD in goodness-of-fit is limited on the level of the overall discrepancy estimation. In contrast, the HD-Explain zooms in the higher granularity by looking at model-conditioned kernel $\kappa_\theta$ among each pair of data, which is further used as an intrinsic score of data influence. We delve deeper into the composition of KSD with greater granularity, focusing on the pairwise relationships between data samples through the Stein Kernel that were never explored in the previous work.

Evaluation Setup:

Evaluating the performance of a post-hoc sample-based explanation approach has been a long-lasting challenge in the literature. Previous research [3] often falls back to qualitative evaluation only by showing images as pos/neg influencers to the prediction without further quantitative analysis. Such an evaluation method is subjective and may lead to disagreement among human auditors. To complement such a shortage, some works [4] may use data debugging metrics as a side quantitative support to show the performance of their proposed explanation approach. However, those methods are questionable in terms of whether they quantify the performance of the explanation or data debugging ability (a good data debugging approach is not necessarily a good prediction explanation approach). We aim to avoid falling back on qualitative evaluations and seek a more straightforward alternative to data debugging. Since the sample-based explanation is essentially a ranking problem (where the explanation algorithm should return a ranked list of training samples as high influencers), we adapt classic information retrieval metrics (e.g., hit rate and coverage), which offer a more robust evaluation by eliminating uncontrolled variables that could affect the results.

Rational of Including noise injection in the experiments:

While the setting looks trivial with the first glance, it turns out that existing solutions (the well-known baselines) produce only 10% - 50% coverage -- many test points receive the same set of explanations, disregarding their unique characteristic. This interesting observation suggests the weakness of the existing approaches on such simple cases where it can happen in practice -- find typical cases with exactly the same symptoms as the new case (in a noise level difference).

Thank you for your feedback. We glad that we have addressed partial of your concerns.

To further address your comments about the comparing HD-Explain with Data modelling approaches, we made further analysis as stated below, where we included our preliminary experimental results (which will be included in the revision) for your reference.

Prediction Explanation vs Data Modelling

Objective Difference:

We would like to restate the difference between prediction explanation and data modelling.
Prediction explanation is to explain the prediction of test data point $x_{\text{test}}$ given a trained model (e.g. a classifier) $f_{\theta^*}(x)$, where the model parameter ${\theta^*} = {\theta^*}(S)$ is trained on training dataset $S$ and fixed during inference time and explanation time. Specifically, the explanation algorithm takes an input test data point $x_{\text{test}}$ and a trained model as input and produces a prediction explanation for $\hat{y}=f_{\theta^*}(x_{\text{test}})$.

Data modelling approaches (e.g. Datamodels and TRAK), on the other hand, do not work on explaining a trained model. Instead, it needs either a parameterized function family $f_{\theta}$ (where $\theta$ denotes free parameters that are not trained). Using Datamodels as an example

Why not using LDS as a metric:

As mentioned by the reviewer, the LDS is a metric for data attribution estimation, not for model prediction explanation specifically. We carefully analyzed the feasibility of including this metric but realized the substantial difference between the two research fields. We have updated our paper draft and included a new section (Section K) to discuss the difference. However, to respectfully address the reviewer’s comment regarding this metric, we followed the LDS evaluation approach proposed in the paper and conducted an experiment on the CIFAR-10 dataset. The average and standard deviation of the LDS are presented in the table below. We have conducted several rounds of experiments to ensure that LDS is not a reliable evaluation metric and demonstrate its variation due to the randomness in sampling the subsets of the data. Here is the provided result for the experiment on the CIFAR10 dataset .

While the metric shows our HD-Explain does better than the baselines, we are not confident with this results given too much variables in the LDS metric, including retraining multiple models on subset of the training dataset. It introduced much noise and making this metric highly unreliable for evaluating prediction explanation algorithms. In particular, from the provided results, we can observe that LDS is notoriously unstable. It requires heavy stress test (large number of subset samples, large number of test sets and ) to ensure its soundness (as proposed in the original paper [1] estimating accurate linear predictors requires thousands of samples, and they conducted the experiment on 2000 test sets for 100 subsets of data)

[1] Park, Sung Min, et al. "TRAK: Attributing Model Behavior at Scale." International Conference on Machine Learning. PMLR, 2023.

Thank you very much for your feedback. Below is our response to the concerns and questions you raised.

Novelty:

We would like to highlight the novelty of this paper as follows:

1. HD-Explain is the very first training-sample-based explanation method that 1) does not leverage model parameter perturbation so far in the literature and 2) shows intuitively understandable explanation samples (as shown in Figures 3, 4, 8).
2. HD-Explain is the very first explanation method leveraging classic statistical discrepancy with kernel tricks. While the extension of Stein discrepancy can be traced back to 1970 and has recently been used for goodness of fit, the statistical tool has never been explored to explain machine learning models, as we showed in paper Section 2.1. Adapting KSD for explanation purposes is a non-retrieval task where it zooms in on the pairwise training data that is never examined in the goodness of fit literature.
3. Standard KSD in the goodness of fit literature is commonly limited to the generative model, which focuses on examining the fitness of $P(\mathbf{x})$. In HD-Explain, we made substantial adjustments to adapt it to consider a whole data point $P(\mathbf{x}, y)$ by re-defining the corresponding kernels.
4. HD-Explain is a prediction explanation method; the research field is very different from the goodness of fit literature. As the reviewer stated in the next question about the rationale of having KSD in the explanation, the idea (Explanation with KSD) behind HD-Explain is not trivial.

How KSD support model explanation:

The great advantage of KSD is the direct calculation of Discrepancy between discrete (data q) and continuous (model p) distributions by the finite sum of kernels. It quantifies the disparity between the two distributions as an expectation of Stein kernels $\mathbb{S}(q,p) = \mathbb{E}_{x,x'\sim q} [\kappa_p(x,x')],$ where the Stein operator augmented kernel $\kappa_p$ uniquely defines pairwise data correlation within the context of a trained model, which can be viewed as a data mesh of the trained model. In other words, the kernel $\kappa_p$ is the model-augmented data correlation that can reflect the tie between a pair of data points. This observation inspired the proposed HD-Explain. By leveraging this property, we can achieve two objectives: 1) pinpoint a subset of training data points that offer the best predictive support for a given test point and 2) detect potential distribution mismatches among the training data points. We believe this is a valuable point to be included in the paper. Hence, we have updated our paper draft by including more discussion about this topic. Please see section 4.4 on the revised paper draft.

Possibility of using other discrepancy definition:

KSD has unique property that allow calculation of discrepancy between a discrete and continuous distribution where the normalization factor for the continuous distribution is not feasible to calculate. In terms of not using MMD, we note that the samples from the continuous distribution are not available which is the requirements for the MMD. In fact, KSD can reduce to a MMD under certain conditions as described [1]. Other well-known diversity definitions, such as KL divergence, are not applicable in measuring the discrepancy between data and modeled distributions since they do need prior knowledge of the data or modeled distribution in a well defined form, which is not accessible here. E.g. Gaussian.

Include Algorithm:

Thanks for this great suggestion, we have updated our paper draft and included a algorithm section in appendix (Section L) for your review.

Dear Reviewer,

Thank you for your feedback on our manuscript. We believe we have addressed all your comments. With the rebuttal deadline approaching, we would greatly appreciate a discussion regarding our responses. Please let us know if there are any points that require further clarification.

Authors