Benchmarking Deletion Metrics with the Principled Explanations

Response to Revewer wwkN

We appreciate the reviewer's acknowledgement of our work. We address the concerns as follows.

[Relevance of Deletion/Insertion Metrics and Their Conformity to What Humans expect]

Deletion metrics are the most popular evaluation metrics of attribution methods because of its conformity to humans intuition, while also generating the evaluation scores of explanations via interactions of the black-box models being explained. Specicially, an attribution explanation present straightforward scores of input features, representing the levels of relevance/importance/sensitivity of the features w.r.t. the black-box model's prediction. Therefore, it is natural to evaluate the quality of the explanations by inspecting "How would the black-box model perform without these explanations?" Furthermore, note that due to the correlations and nonlinearities of features, deleting $k$ features and deleting $k+1$ features do not lead to additive predictions of the black box. Therefore, the progressive deletion of features are preferred for it explores all possible $k$s.

We also present additional qualitative analysis of TRACE -- the principled explanation of the deletion metrics, to demonstrate how progressive deletion works in finding features that affect the model's performance the most in the deletion scenario. The analysis can be found in Appendix G, where we carefully inspect the visualized deletion trajectories solved by TRACE.

[Rankings of Baselines]

The proposed TRACE framework aims at benchmarking the most popular deletion metric for explanation methods rather than the explanation methods themselves, and thus the analysis and the conclusions were not focused on specific existing explanation methods.

Using the benchmarked variants of the deletion metrics, we can have the following rankings of existing explanations under the verified deletion metric. The rankings illustrate "to what extend can the selected features of these explanation methods affect the model's behavior":

• Probability: GradCAM (GC) $>$ Information Bottleneck Attribution (IBA) $>$ RISE $>$ Excitation Back-Propagation (EBP) $>$ RISE $>$ Extremal Perturbation (EP) $>$ Gradient (Grad) $>$ Integrated Gradient (IG) $>$ Input $\times$ Gradient (IxG)
• Logits: GradCAM (GC) $\approx$ RISE $>$ Information Bottleneck Attribution (IBA) $>$ Excitation Back-Propatation (EBP) $>$ Extremal Perturbation (EP) $>$ Gradient (Grad) $>$ Integrated Gradient (IG) $>$ Input $\times$ Gradient (IxG)

We have the following remarks on the results

(1) The results suggest that (a) GradCAM is the state-of-the-art among all existing attribution methods for selecting features that maintain the model's performance when kept and harm the model's performance when deleted; and (b)perturbation-based methods (eg. RISE, EP, etc.), although related more closely to feature perturbations, do not have significant superiority over back-propagation-based method (GradCAM, EBP, Gradient, etc.) under the deletion metric. In fact, (3) it is noticed that pixel-wise attributions (e.g. IG, IxG and Gradient) tend to underperform the deletion metrics. It is the feature size (pixel-level, patch-level, etc.) that affect more in the deletion metric rather than the difference between back-propagation and perturbation.

(2) Therefore, we advocate that the metrics for explanations should serve specific and practical purposes. For instance, deletion metrics measure "to what extent can the selected features of these explanation methods affect the model's behaviour", Invariance/Equivariance Robustness [1] measures whether the highlighted features are invariant/equivariant w.r.t. transformations, etc.

(3) By finding the optimal explanation of the deletion metrics, we also emphasize that the evaluation of existing explanation methods should not be based on a single and universal metric. Failing certain metrics does not instantly disprove explanation methods (e.g. gradient in deletion test). It's also likely that the metrics and the explanation methods simply focus on different goals.

[Layerwise Relevance Propagation (LRP)]

We thank the reviewer very much for mentioning the LRP methods. In fact, the rule-based back-propagation explanation method LRP [2] is closely related to gradient-based methods. Proposition 3 in [3] demonstrates that $\epsilon$-LRP is equivalent to Input $\times$ Gradient (IxG) under ReLU nonlinearities. In order to have a more comprehensive demonstration, we include more attribution methods such as LRP, guided GradCAM, gradient SHAP, etc. in the comparison. The results have been updated to the appendix G.

[Blurring Reference Implementation]

Let $\mathbf{x}\in\mathbb{R}^D$, the deletion of features is implemented as $x_{deleted}=\mathbf{x}\odot\mathbf{m}+\mathbf{r}\odot(1-\mathbf{m})$ where $\mathbf{r}\in\{0,1\}^d$ is the binary mask determined by the trajectory/heatmap/number of features to be deleted. And $\odot$ is the Hadarmard product that performs feature-wise deletion. $\mathbf{r}\in\mathbb{R}^d$ is the reference values. In practice, we change the reference values through the definitions of $\mathbf{r}$. That is, $\mathbf{r}\_{zero}=\mathbf{0}, \mathbf{r}\_{mean}=\bar{x}\mathbf{1}$. As for blurring, we use $\mathbf{r}\_{blur} = `Blurrer`(\mathbf{x})$, where $`Blurrer=torchvision.transforms.GaussianBlur(kernel\\_size=(31, 31), sigma=(56, 56))`$.

[Fig 2a Illustration]

We thank the reviewer for the suggestions! We would first want to elaborate that there are no -GO curves because it is unsolvable. In Figure 2a, we demonstrate how well the TRACE solved by simulated annealing can approximate the global optimum (-GO) by approaching the unreachable bound of complete search (CS). And the global optimum is hence squeezed between them, as the Cyan and Lime color regions. The fact that these two regions are too slim to recognize actually verifies that TRACE-SA is able to approximate the theoretical global optimum very closely compared with the large AUCs.

[Elaborations on Table 1]

We thank the reviewer for pointing out the ambiguity in the analysis of table 1.

• Since the ratio $\frac{\textrm{variance of TRACE}}{\textrm{improvement from TRACE}}$ is too small, we do not repeat the experiments for the same (model, reference, input) tuple. Also, it should be noted that back-propagation-based explanation methods are also deterministic. As a result, for the (model, reference) pair, we report the mean LeRF-MoRF score over samples.

• For the analysis of TRACE and the metric, the analysis of the results shown in table 1 can be found in sections 5.1, 5.2. Here we summarize them as follows: (1) In each row, we verify the TRACE's optimality. (2) In the TRACE-p and TRACE-y columns we demonstrate that the difference between "using probability" and "using logits" as the indicator is not negligible and should be decided carefully w.r.t. the evaluation purpose. (3) Comparing the three blocks with different reference values, we conclude that with the selected settings, the differences caused by reference values are almost negligible.

• For the analysis of existing explanation methods, we can also draw insightful conclusions. Please see the response in [Rankings of Baselines].

[Details in Notions, References and Grammar]

We thank appreciate the reviewer's careful and detailed suggestions! We've revised the mentioned notions, typos and grammar mistakes.

References

[1] "Evaluating the Robustness of Interpretability Methods through Explanation Invariance and Equivariance", NeurIPS 2023

[2] "On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation", PloS one, 10(7), e0130140.

[3] "Towards better understanding of gradient-based attribution methods for deep neural networks", ICLR 2018

[Q2: Why the optimization of MoRF or LeRF can be considered as explanation-method-agnostic approach?]

We would like to clarify that TRACE is considered as an "explanation-method-agnostic" study of the deletion metric $M_{f,x}$ because it investigates the metric through its optimizer $\psi^*$, differing from existing studies that rely on attribution explanations $\psi_{IG},\psi_{Gradient},\psi_{GradCAM},\cdots$. Specifically, these explanations are independent of the metric $M_{f,x}$, and using $M_{f,x}(\psi_{IG}),M_{f,x}(\psi_{Gradient}),M_{f,x}(\psi_{GradCAM})$ to evaluate the metric itself can be greatly biased by these explanations. Oppositely, the optimizer $\psi^*$ is determined solely by the metric $M_{f,x}$. Using $M_{f,x}(\psi^*) = max_{\Psi}M_{f,x}$ as the pipeline to evaluate the metric is not biased by any existing explanation methods, and thus is considered explanation-method-agnostic.

On the other hand, we would clarify that although the optimizer is referred to as "the principled explanation", it is completely derivative from the metric itself, and should not be confused with existing explanations.

[The studies of Deletion Metrics in [Ref3]]

The deletion metric is referred to as the Area Over the Perturbation Curve (AOPC) in [Ref3]. The work explores the AOPC scores of multiple attribution explanation methods such as SHAP, IxG, etc., and under MoRF/LeRF, respectively. [Ref3] fall short in a similar manner as [1,2] to rely on these explanation methods. Without the principled explanation that illustrates what the metric is really measuring, [Ref3] mainly serve as a sanity check for the deletion metric, as they point out the fact that "MoRF and leRF are highly-variable across saliency methods" but do not provide mitigations for the issue.

[Q3: New Findings Comparing with Related Work [Ref1] & [Ref2]]

First, we would emphasize that both [Ref1] and [Ref2] do not optimize the deletion metric (i.e. MoRF, LeRF, LeRF-MoRF), nor do they optimize any smooth or relaxed version of the deletion metric. We now review the two references and compare our work with them in details. Note that the proposed two perturbation-based methods share many common characters, so we begin by discussing the shared differences. Then we delve into the differences between them and TRACE separately.

1. General Review of [Ref1, Ref2]

[Ref1] and [Ref2] both aim at generating relaxed masks so that the prediction of the masked input is minimized and are perturbation-based explanation methods. Both of them use a single continuous mask $\mathbf{m}\in[0,1]^d$ to perturbe the input image through convex combination $\bar{\mathbf{x}}_{m,r} = (\mathbf{1}-\mathbf{m})\odot\mathbf{x} + \mathbf{m}\odot\mathbf{r}$. And they aim at solving

• [Ref1]: $\mathbf{m}^*=\arg\min_{[0,1]^d}\lambda\|\mathbf{1}-\mathbf{m}\|\_1+f(\bar{\mathbf{x}}_{w,r})$. (eq (3) of [Ref1])
• [Ref2]: $\mathbf{m}^*=\arg\min_{[0,1]^d}\sum_{i=1}^d(1-m_i)+\lambda\mathbb{E}[f(\bar{\mathbf{x}}_{w,r})]$. (eq (3.2) of [Ref2])

(In order to universally review [Ref1,Ref2], we unify the notations of the two work.)

It can be observed that the only difference is that [Ref1] use pre-defined references while [Ref2] use the expectation over random noise references. On the other hand, TRACE, aiming at optimizing the deletion metric, is defined as follows:

• TRACE-Mo: $\tau^*=\arg\min_{\tau}\sum_{k=0}^df(\mathbf{x}_{\tau[:k]})$

We can observe that there's almost no similarity between TRACE's objective (the deletion metric) and [Ref1,Ref2]. e.g.,

(1) [Ref1,2] solve for a one-time deletion mask that minimize the model's prediction once applied, but TRACE follows the deletion test to include all possible number of deleted features to optimize the progressive deletion process.

(2) Therefore, [Ref1,2] are only able to guarantee the performance of feature deletions with a fixed number of deleted features, while TRACE guarantees all.

(3) In fact, [Ref1,2] are more similar to the smooth/relaxed version of a single term of the complete search (CS) in our manuscript. In complete search, a single term can be seen as a minimization with $L_0$ norm on the mask (fixed number of features perturbed).

(4) The relaxation enables differentiable optimization, but even for their own objective, the optimality has not been rigorously studied, indicating that they might fall into local optimum and there are better masksl, let alone the deletion metric. As a principled explanation, TRACE's optimality for the deletion metric has been verified through the squeezing between the complete search.

(5) The relaxation of the masks make the explanations proposed by [Ref1,2] intrinsically incompatible with the deletion metric, where features are deleted with strict binary masks in each term.

etc.

In summary, the only common point might be they all aim at finding a way to minimize/maximize the models prediction. But that holds true for all perturbation-based methods.

[The Outline of the Response]

We thank the reviewer for the insightful comments and important references. Here we address the reviewer's two main concerns in the following manner:

1. [The Validity and the Purpose of Optimization, and the Explanation-Method-Agnostic Claim of TRACE] This point is connected to the First two questions brought up by the reviewer. Therefore, we address the posed questions by answering how the existing problems are resolved by TRACE.

2. [Additional Related Work] After addressing the concern regarding the validity, we carefully review the related work mentioned by the reviewer. We review [Ref3] first and then answer Q3 by reviewing [Ref1,2]. These important related works have also been cited properly in our updated manuscript.

[Q1: How is the optimization of MoRF or LeRF relevant to the investigation of their properties?]

1. The optimum of a metric illustrates what the metric expects from the candidates (explanation methods) being measured.

If the best- or very-high-performing explanation (principled explanation) under a certain metric does not make sense, it indicates such a metric is improper. We draw an analogy to evaluation metrics in classification. In highly imbalanced data, directly optimizing (i.e. searching) for a classifier that achieves very-high-performing accuracy score is likely to result in a model that blindly classifies all samples into the majority class. And thus we can verify that the accuracy metric is improper in this scenario.

2. A Formalized Elaboration

We elaborate on this using the following formulation. Given a black-box prediction $f(\mathbf{x})$, a metric can be seen as a mapping $M_{f,x}:\Psi\rightarrow \mathbb{R}$, where $\Psi$ is the set of all explanations w.r.t. the prediction $f(\mathbf{x})$. In order to study whether the metric $M_{f,x}$ is legitimate, TRACE aims at finding the optimizer $\psi^* = \arg\max_{\psi\in\Psi}M_{f,x}(\psi)$ of the metric, i.e., the "principled explanation" that optimizes the deletion metrics. Such principled explanation $\psi^*$ intrinsically reviews the deletion metric and represents "what the metrics are expecting from explanations".

3. The Problem of Studying the metric $M\_{f,x}$ without its optimizer $\psi^\*$

Existing studies of metrics highly rely on explanation methods. They inspect the metrics by checking the scores obtained by well-accepted explanation methods. For instance, [1,2] rely on popular methods such as IG, Gradient, etc. to investigate deletion metrics. This is equivalent to checking their scores $M_{f,x}(\phi_{IG}),M_{f,x}(\phi_{Gradient}),\cdots$, which can be problematic for the following reasons:

(1) Since deletion metric $M_{f,x}(\psi) = \sum_{k=0}^df(\mathbf{x}\_{\sigma(\psi)[:k]})$ (using MoRF as an example) is infeasible to study analytically, enumerating these explanation points $\phi_{`explanation`}\in\Psi$ does not help in exploring the property of the metric $M_{f,x}$ because there's no way to tell if their scores are high or low.

(2) Since $M_{f,x}$ is discontinuous, even the relative score $M\_{f,x}(\psi\_{`explanation1`})-M\_{f,x}(\psi\_{`explanation2`})$ between two explanations provide no insights in understanding the metric.

As a result, the optimizer $\psi^*$ of $M_{f,x}$ solved by TRACE might be the only property of the metric that we can have access to. And we study the deletion metric $M_{f,x}$ through this optimizer to investigate whether the metric $M_{f,x}$ is legitimate itself.

4. An Example of the Use of the Optimizers

By revealing the principled explanation, we explore the connection/conflict between different metrics, by answering the question "Does the best-performing explanation under one metric work well in other metrics"? For example, in Section 5.2, we studied the connection/conflict between different deletion criteria, MoRF and LeRF, resulting in two metrics $M^{MoRF}\_{f,x}$ and $M^{LeRF}\_{f,x}$ with optimizers $\psi^{MoRF}, \psi^{LeRF}$. We thus employ $M^{LeRF}\_{f,x}(\psi^{MoRF})$, $M^{MoRF}\_{f,x}(\psi^{LeRF})$ to inspect whether these metrics are legitimate.

Here, we draw an analogy to another important topic in trustworthy AI: fairness, where the connection/conflict among fairness notions has been widely studied [3,4]. There is an intrinsic conflict between two common group fairness metrics: demographic parity (DP) and equalized odds (EOd), since a classifier with the best EOd result may not ensure the best DP result. Also, a classifier with the best accuracy may not ensure the best DP result. A similar discussion on the connection/conflict among evaluation metrics in XAI is crucial yet underexplored, partly due to the challenge of identifying optimal explanations. TRACE enables us to systematically investigate this aspect.

[Q3: New Findings Comparing with Related Work [Ref1] & [Ref2]]

2. Details of [Ref1]

[Ref1] indeed point out that deleting features to minimize the models' prediction can create artifacts, but [Ref1] mitigate this by introducing stochasticity in $m$ and enforcing a total-variation normalization. Although they are effective in generating the perturbation-based attribution method, they provide no help in mitigating the OOD problem in the deletion metric whatsoever.

TRACE, on the other hand, is developed to benchmark the most popular and widely used deletion metric. Thus almost all the conclusions regarding the metric that are drawn from our work by TRACE differ from [Ref1]. For instance, even for the artifacts discussed in both works, TRACE suggests that they are inevitable in the progressive deletion with small patch sizes, but using LeRF$-$MoRF or LeRF instead can mitigate the issue greatly. [Ref1], though describing the "preservation game" very briefly in section 4.2, do not provide any further analysis of its effectiveness.

3. Details of [Ref2]

[Ref2], although similar to [Ref1] as a perturbation-base method, do carry out experiments testing the proposed methods with the deletion metric and verify that [Ref2] outperform the selected explanation methods. However, it falls short in the following aspects.

(1) The optimality is important for the principled explanations to benchmark the metrics.

Although [Ref2] outperform the selected attribution methods in the deletion test, there's no guarantee for the optimality theoretically (e.g. optimization objective) or empirically (e.g. the comparison between CS to quantify -GO). Therefore, it only demonstrates that "[Ref2] are better than these selected explanation methods", or formally $M_{f,x}(\psi_{`[Ref2]`}) > M_{f,x}(\psi_`selectedExplanations`)$. Recall that in the previous analysis of Q1, point 3, the relative difference of scores does not have any insights due to the discontinuity of $M_{f,x}$. There's no guarantee whether the next selected explanation methods will outperform them.

Besides, it is noticed that the selected methods are those that are verified to be inconsistent with the deletion metrics in our manuscript. Differently, TRACE is guaranteed to outperform all attribution explanation methods in the deletion metric. This enables us to perform rigorous studies of the deletion metric by checking the TRACE results.

(2) Both Directions Matter

By combining LeRF and MoRF, we conclude that although LeRF is much better than MoRF and is less affected by the OOD issue, LeRF$-$MoRF is actually the best criterion for the deletion metric, and outperforms LeRF by a large margin. The corresponding TRACE-Le$-$Mo provides the principled explanation. However, from the optimization objective, it can be noticed that [Ref2] can not solved for combined directions.

(3) Studies Focusing on the Deletion Metrics

Using the optimizer $\psi^*$, we carry out a rigorous study over the question "What are the metrics really expecting from the explanations?", and propose many practical guidelines on how the deletion metrics should be used and how the variants are selected. These regulations on how to evaluate explanation methods are important but missing from these references.

[Typo of Proof of Theorem 3.1]

We thank the reviewer for pointing this typo. In the proof of theorem 3.1, all the terms in eq (5) should be $f(x)$ instead of just $x$. In fact, given a trajectory $\tau$, for deletion with MoRF, the AUC is written as $f(x_{\backslash\tau[d:]}) + f(x_{\backslash\tau[d-1:]})+\cdots+f(x_{\backslash\tau[0:]}) = \sum_{k=0}^df(x_{\backslash\tau[k:]})$. Similarly, for insertion with LeRF, the AUC is written as $f(x_{\tau[:0]})+f(x_{\tau[:1]})+\cdots+f(x_{\tau[:d]}) = \sum_{k=0}^df(x_{\tau[:k]})$. Since deleting the least important $k$ features is equivalent to inserting the most important $d-k$ features, we have $\sum_{k=0}^df(x_{\backslash\tau[k:]})=\sum_{k=0}^df(x_{\tau[:k]})$.

References

[1] "A Benchmark for Interpretability Methods in Deep Neural Networks", NeurIPS 2019

[2] "A consistent and efficient evaluation strategy for attribution methods", ICML 2022

[3] "FACT: A diagnostic for group fairness trade-offs", ICML 2020

[4] "Translation tutorial: 21 fairness definitions and their politics", Proc. conf. fairness accountability transp., new york, usa. Vol. 1170. 2018.

[Ref1] "Interpretable Explanations of Black Boxes by Meaningful Perturbation", ICCV 2017

[Ref2] "Feature Attribution As Feature Selection", Openreview 2018

[Ref3] "Sanity Checks for Saliency Metrics"

We thank the reviewer very much for pointing out this very recent related work. We will address the reviewer's concern by explaining the difference between our work and [Ref3].

1. The optimality is still not guaranteed in [Ref3].

As we've agreed that optimality is essential for the principled explanation of a metric, we first discuss how [Ref3] compromise in achieving the optimality.

(1) As shown in Algorithm 1 of [Ref3], beam search is used to search for a solution. In each iteration, beam search extends to find the next feature to add to the solution $e^*$. Such a step-by-step search is in fact a relaxed greedy scheme that does not take the entire trajectory into consideration. Therefore, it suffers from all the drawbacks of the standard greedy search that simply searches for the next feature to insert/delete to maximize/minimize the prediction of the next step.

(2) The justification for the use of beam search is technically deprecated.

It is mentioned in [Ref3] that the beam search is effective in optimizing the objective "because it's 'additive'" on Page 5, left column, under Algorithm 1. The "additive property" is defined in Definition 4.1 that if a metric $m(x,e)$ (where $e$ is the explanation under their notations) can be written as a summation $m(x,e)\sum_{l=1}^L h(x, e^{(l)})$ for sum $h$, where "$e^{(l)}$ reveals the attribution values of $l$ most important features according to $e$ but keeps the rest inaccessible."

It should be noted that these arguments are technically flawed because $e^{(l)}$ and $e^{(l+1)}$ are never independent, let alone additivity. On the contrary, they are actually incremental and highly correlated. Hence greedily searching for the next step does not optimize the entire targeted AUCs.

(3) Since the optimality is not supported from the theoretical perspective, [Ref3] is essentially equivalent to [Ref2] in finding an explanation method that outperforms the selected existing explanation methods under the deletion metric, but the optimality is never guaranteed. As a result, [Ref3] is not the principled explanation. Without the optimality, [Ref3] suffers from the exactly same issue as [Ref2] as shown in point 3 of the review of [Ref2]:

• Since deletion metric $M\_{f,x}(\psi) = \sum\_{k=0}^df(\mathbf{x}\_{\sigma(\psi)[:k]})$ (using MoRF as an example) is infeasible to study analytically, enumerating these explanation points $\phi_{`explanation`}\in\Psi$ does not help in exploring the property of the metric $M_{f,x}$ because there's no way to tell if their scores are high or low.

• Since $M_{f,x}$ is discontinuous, even the relative score $|M_{f,x}(\psi_{`explanation1`})-M_{f,x}(\psi_{`explanation2`})|$ between two explanations provide no insights in understanding the metric.

(3) As a greedy algorithm, beam search is also short-sighted and it thereby cannot include both directions (MoRF and LeRF) together. As we've verified in our work LeRF is only a compromise to LeRF$-$MoRF.

(4) In contrast, TRACE treats the solution trajectory as a whole and makes use of combinatorial optimization schemes so that all the issues above are resolved. For example, (a) Instead of just the several selected baseline explanation methods, TRACE is guaranteed to outperform all attribution explanation methods in the deletion metric. (b) Solving for the entire trajectory $\tau$ enables us to optimize the objective LeRF$-$MoRF directly. (c) Also, the true optimality of TRACE-SA is demonstrated by comparing it with the theoretical global optimum (TRACE-GO) and Complete Search (CS). etc.

2. Benchmarking the Deletion Metrics

In conclusion, [Ref3] aims to develop an explanation method that achieves better scores under the deletion metric by searching the feature to delete step-by-step. The discussion is also mainly focused on using it as an explanation while the objective, the deletion metric itself is not studied.

Differently, TRACE is proposed to benchmark all the variants of deletion metrics to resolve the potential problems of the metric such as the infamous OOD issue, so that this favorable metric of attribution explanations can be used carefully and normatively. As we have agreed, rigorous studies and analyses of this nature should be conducted using principled explanations $\psi^*$, a domain that is only achievable with TRACE and has not been explored previously. The application of such a theoretically and empirically validated optimal trajectory as a method for deletion-wise explanation not only fulfills this criterion but also adds significant value to the field.

References

[Ref2] "Feature Attribution As Feature Selection", Openreview 2018

[Ref3] "The Solvability of Interpretability Evaluation Metrics", EACL 2023 (Findings)

[Correlated Fatures]

Due to the highly non-linearity of DNNs, features are usually correlated in the forward pass. However, this nonlinearity makes it impossible to study the behaviors of models. As a result, locally linear/additive/decoupling assumptions are usually made in the studies of the mechanism of DNNs, especially in attribution methods. As such, the deletion/insertion metric itself makes this compromise, too, where features are deleted/inserted independently. That is, given an attribution map $\phi_f(x)\in R^d$, even if the ith and the jth features are correlated, the input features are still perturbed independently based solely on the relation between $\phi_f(x)_i$ and $\phi_f(x)_j$. The correlated features, when deleted independently, can lead to confusion, because the order of their deletions matters. Fortunately, TRACE takes all possible deletion trajectories into consideration and solves for the optimizer trajectory, which mitigates the influence of the ordering of correlated features.

We present an example of correlated features. In order for the ground truth to be known and controllable, suppose $f(x)=x_1+ax_2+x_3+2x_1x_2, x=(1,1,1)$, where $x_1,x_2$ are correlated. Given a trajectory $\tau$ of the 3 features, let $s(\tau)$ be the corresponding LeRF$-$MoRF scores under the deletion metric. A higher score indicates a better trajectory under the deletion metric.

We can see that using TRACE, the trajectories are found in the following ways:

(i) when $a=1$, $x_1,x_2$ are symmetric. The LeRF-MoRF scores for the 6 trajectories are shown in the third row. Both $\tau=[3,1,2]$ and $\tau=[3,2,1]$ are optimal, suggesting that $x_1,x_2$ are equivalently important in the deletion metric.

(ii) when $a< 1$, $x_1$ is more important. In such case, we have in the top two rows that $\tau=[3,2,1]$ has the highest score. This verifies that TRACE recognizes $x_1$ as the most important one (last to delete).

(iii) when $a>1$, similarly, $x_2$ is more important. And from the last two rows, $\tau=[3,1,2]$ has the highest score. This verifies that TRACE recognizes $x_2$ as the most important one, too.

This example shows that based on the prediction change w.r.t. the deletion metric, TRACE is able to detect the relative importance of the correlated features to the model recognized by the deletion metrics.

Additionally, for image datasets, it should be noted it's a general consensus that pixels that are spatially close usually tend to have higher correlations. Therefore, as TRACE suggested, casting deletions/insertions w.r.t. the superpixel patches instead of the small pixels individually, is already an approach to take the correlation of features into consideration.

[Tabular Datasets]

Based on perturbing input features, the optimality of TRACE is independent from the data modality, making it readily applicable to other domains. We've added experimental results on tabular data in appendix G, along with more detailed analysis. The results are consistent with image data.

[Insights of model architectures and sizes discovered by TRACE]

In fact, we can observe quantitative insights between model structures and model sizes. We've added experiments in appendix H to demonstrate how TRACE transfers

Since TRACE trajectories illustrate the intrinsic influence of the feature deletion towards the models performance, the similarities between model structures and sizes can be quantitatively through the transferability of the trajectories. That is, "Will the model $f_2$ achieve the same level of AUCs following the optimal trajectory from model $f_1$?" The results have been added to the appendix, where we've verified the similarities between different sizes from same model families (e.g. mobilenet, resnet) and between different model families.

References

[1] "Evaluating the Robustness of Interpretability Methods through Explanation Invariance and Equivariance", NeurIPS 2023

[W2: Claims over Existing Explanation Methods]

The proposed TRACE framework aims at benchmarking the popular deletion metric for explanation methods using the principled explanation -- without relying on any existing explanation methods. Thus the analysis and the conclusions were not focused on analyzing specific existing explanation methods. But with the benchmarked variants of the deletion metrics, we sure can have the following rankings of existing explanations under the verified deletion metric.

(1) The ranking illustrates "to what extent can the selected features of these explanation methods affect the model's behavior":

• Probability: GradCAM (GC) $>$ Information Bottleneck Attribution (IBA) $>$ RISE $>$ Excitation Back-Propagation (EBP) $>$ RISE $>$ Extremal Perturbation (EP) $>$ Gradient (Grad) $>$ Integrated Gradient (IG) $>$ Input $\times$ Gradient (IxG)
• Logits: GradCAM (GC) $\approx$ RISE $>$ Information Bottleneck Attribution (IBA) $>$ Excitation Back-Propatation (EBP) $>$ Extremal Perturbation (EP) $>$ Gradient (Grad) $>$ Integrated Gradient (IG) $>$ Input $\times$ Gradient (IxG)

The results suggest that (a) GradCAM is the state-of-the-art among all existing attribution methods for selecting important features that maintain the model's performance when kept and harm the model's performance when deleted; and (b)perturbation-based methods (eg. RISE, EP, etc.), although related more closely to feature perturbations, do not have significant superiority over back-propagation-based method (GradCAM, EBP, Gradient, etc.) under the deletion metric. In fact, (3) it is noticed that pixel-wise attributions (e.g. IG, IxG and Gradient) tend to underperform the deletion metrics. It is the feature size (pixel-level, patch-level, etc.) that affect more in the deletion metric rather than the difference between back-propagation and perturbation.

(2) Therefore, we advocate that the metrics for explanations should serve specific and practical purposes. For instance, deletion metrics measure "to what extent can the selected features of these explanation methods affect the model's behavior", Invariance/Equivariance Robustness [1] measures whether the highlighted features are invariant/equivariant w.r.t. transformations, etc.

(3) By finding the optimal explanation of the deletion metrics, we also emphasize that the evaluation of existing explanation methods should not be based on a single and universal metric. Failing certain metrics does not instantly disprove explanation methods (e.g. gradient in deletion test). It's also likely that the metrics and the explanation methods focus on different goals.

We thank the reviewer for the suggestions and questions. We will address all of them in detail.

[W1: Presentations]

We address the reviewer's concern point-by-point.

1. [Intuitions on Proofs]

Our work presents three main proofs. They demonstrate the following facts:

• B.2/B.3: We show that the attribution maps can be simplified to trajectories with mathematical rigors. This is done by showing that the trajectories can be seen as equivalence classes of attributions. The intuition behind this is clearly stated in section 2. Trajectory Importance (TRACE), page 3 of the manuscript.

• 3.1: We show that we distinguish between MoRF and LeRF, then there's no need to distinguish between deletion and insertion anymore, because deletion-MoRF is equivalent to insertion-LeRF, and deletion-LeRF is equivalent to insertion-MoRF. The intentions and intuitions are clearly discussed in section 3, page 4, the Deletion vs. Insertion paragraph.

• 4.1: We show that the optimization problem for TRACE is NP-hard. This is to justify the use of combinatorial optimization tools for solving the optimization problem. The intuitions for the proof are discussed in the first paragraph of section 4.

We believe both the statements and the proofs are clear to follow. We would appreciate it if the reviewer could point out the part where the proofs may appear obscure or misleading so that we can improve the presentation of the proofs accordingly.

2. [Related Work]

We acknowledge the reviewer's concern regarding the placement of related work. We would clarify that all the important related work (i.e. the studies of the explanation metrics and tested attribution methods) are thoroughly discussed in the introduction section (paragraphs 2-4) of the manuscript. This choice was made deliberately to enhance the reading flow and ensure that essential background information is immediately accessible to the reader.

We assure that the additional related work section in the appendix is not crucial for understanding any arguments and findings of our study. Instead, it serves as an extensive reference list for those interested in a deeper exploration of XAI studies focused on attribution explanations.

It contains two parts, the attribution methods and attribution metrics. The first part elaborates on the wide variety of existing attribution methods, including those we have referenced previously for our experiments. The second part provides an overview of the research landscape concerning attribution metrics, essentially offering a different perspective to the introduction. It transitions from a point-guided manner to a work-guided manner.

3. [Concerns Regarding the Clarity]

We thank the reviewer for their valuable feedback regarding the clarity and formality of certain sections of our manuscript. We take these comments seriously and are committed to enhancing our manuscript where "important specifics" are missing or the contexts are "linguistically difficult" to understand. We'd appreciate it if the reviewer could point out these areas, and we will make targeted revisions to address these concerns.

4. [Notions & Acronyms]

We appreciate the reviewer for mentioning the problems in the notions and acronyms. We've carried out a revision of the manuscript to ensure the notions and acronyms are consistent.

We thank the reviewer very much for the appreciation of our work! Here we address the reviewer's further concerns as follows.

[The insights for given explanation methods]

For an explanation generated by a given explanation method, we can draw the following conclusions:

(1) With the benchmarked variants of the deletion metrics, we can have the following rankings of existing explanations under the verified deletion metric. The rankings illustrate "to what extend can the selected features of these explanation methods affect the model's behavior":

• Probability: GradCAM (GC) $>$ Information Bottleneck Attribution (IBA) $>$ RISE $>$ Excitation Back-Propagation (EBP) $>$ RISE $>$ Extremal Perturbation (EP) $>$ Gradient (Grad) $>$ Integrated Gradient (IG) $>$ Input $\times$ Gradient (IxG)
• Logits: GradCAM (GC) $\approx$ RISE $>$ Information Bottleneck Attribution (IBA) $>$ Excitation Back-Propatation (EBP) $>$ Extremal Perturbation (EP) $>$ Gradient (Grad) $>$ Integrated Gradient (IG) $>$ Input $\times$ Gradient (IxG)

The results suggest that (a) GradCAM is the state-of-the-art among all existing attribution methods for selecting important features that maintain the model's performance when kept and harm the model's performance when deleted; and (b) perturbation-based methods (eg. RISE, EP, etc.), although related more closely to feature perturbations, do not have significant superiority over back-propagation-based method (GradCAM, EBP, Gradient, etc.) under the deletion metric. In fact, (c) it is noticed that pixel-wise attributions (e.g. IG, IxG and Gradient) tend to underperform the deletion metrics. It is the feature size (pixel-level, patch-level, etc.) that affect more in the deletion metric rather than the difference between the back-propagation and perturbation.

(2) Therefore, we advocate that the metrics for explanations should serve specific and practical purposes. For instance, deletion metrics measure "to what extend can the selected features of these explanation methods affect the model's behavior", Invariance/Equivariance Robustness [1] measures whether the highlighted features are invariant/equivariant w.r.t. transformations, etc.

(3) By finding the optimal explanation of the deletion metrics, we also emphasize that the evaluation of existing explanation methods should not be based on a single and universal metric. Failing certain metrics does not instantly disprove explanation methods (e.g. gradient in deletion test). It's also likely that the metrics and the explanation methods simply focus on different goals.

The proposed TRACE framework aims at benchmarking the most popular deletion metric for explanation methods rather than the explanation methods themselves, and thus the analysis and the conclusions were not focusing on any specific existing explanation methods. However, we believe the insights mentioned above are important to the XAI studies.

References

[1] "Evaluating the Robustness of Interpretability Methods through Explanation Invariance and Equivariance", NeurIPS 2023