Unlearning-based Neural Interpretations

We thank the reviewer for their detailed feedback and constructive suggestions.

Re: W1 - We strengthen our discussion of the motivation and technical details behind UNI with a simple motivating example using a gaussian mixture model (Section 4.3 and general response). We explain the necessity of unlearning and activation matching, by illustrating how a naive gradient descent approach (to minimise model confidence) omits more desirable baseline solutions that UNI is able to discover. We observe that UNI preserves path monotonicity, low-curvature and proximity and justify how it satisfies the weak dependence property for conformal path attribution. We further note that UNI has in-depth contributions to broader XAI, and elaborate on this in the general response and revised manuscript. In particular, we are optimistic that UNI’s success will facilitate further research into 1) granular unlearning for hierarchical feature/concept attribution; 2) aligning XAI methods and models; 3) comparing network inductive biases and interpretability during training.

Re: W2 & Q1 - To motivate activation matching, first consider that ReLU networks have decision boundaries that are well represented as piecewise linear functions. First-order unlearning reduces the magnitude of piecewise linear weights that are highly activated by salient features; activation matching discovers a baseline in the image space that replicates this activation pattern. This encourages the baseline and the input to lie in the same connected component, which crucially satisfies the weak dependence property for conformal path-based attribution.

Re: W3 & Q3-5 - We revise Sections 4.2-4.3 to clarify concepts of weak dependence and completeness and how they relate to the quality of the path features (monotonicity, low curvature, proximity). Weak dependence—a smooth transition between the absence and presence of salient features—is crucial for meaningful path attribution, and is only compatible with completeness in the case where the baseline and the input lie in the same connected component (in the case of piecewise-linear models). A path (from baseline to image) with monotonically increasing model confidence suffices to satisfy a weak version of the criteria for valid path features $\textrm{sgn}(\nabla_x F_c(x)) \cdot \textrm{sgn}(\nabla_{\tilde{x}} F_c(x^\prime))=1$ (Akhtar & Jalwana, 2023); a path that exhibits low curvature (i.e. eigenvalues of the hessian) on and in a neighbourhood of the baseline reduces the variability and robustness of the calculated gradients under small-norm perturbations; a specific baseline (enforced by step 5 of Algorithm 1) that lies within an $\varepsilon$-ball of the image satisfies the proximity condition required for weak dependence.

Re: W4 - Fixed and thanks!

Re: Q2 - An unlearned baseline is necessary because it reflects which features are salient for a specific model’s prediction. Static baseline choices (e.g. using constant (black, channel mean) pixel baselines, gaussian blur, random baselines) project assumptions onto the model, which result in undesirable biases that confound interpretation results. Since it is important to attribute features in the pixel space, the suggested baseline could 1) introduce a colour bias that specific pixel values are unimportant for inference, 2) lie far away from the local image neighbourhood and violate the weak dependence property. Unlearning allows for conformal paths from baseline to input, facilitating robust interpretations.

Re: Q6 - To demonstrate that UNI attributions align well with specific task-model-image triads, we perform additional experiments in Appendix A.1. We visually and numerically compare UNI attributions across models for the same image input, with the LPIPS image similarity metric. We observe in Figure 8 that attributions differ significantly and even reflect the inductive biases of respective models (e.g. grid-like artefacts are present in ViT attributions whereas smoother attributions are computed for convolutional architectures). Table 7 scores reflect the dissimilarity between the original image and unlearned baselines (pixel-wise and in terms of model confidence).

We again express our gratitude to the reviewer for their help in improving our work. We look forward to a fruitful discussion!

Thank you for the detailed response, which addresses most of my concerns. I have raised my score from 5 to 6.

We thank the reviewer for their detailed feedback and insightful comments.

1. Major comments
Re: M1 - We thank the reviewer for their valuable suggestions. RISE, Sobol and FORGrad (Saliency strikes back) are indeed very relevant to our work and we discuss their contributions in Section 6, particularly how they improve attribution faithfulness.

Re: M2 - We supplement MuFidelity score results with deletion and insertion score results in Tables 3 and 4, and discuss how they provide an approximately causal view into which pixels are necessary and sufficient for a particular prediction result. We also detail hyperparameters used (for MuFidelity, deletion and insertion scores) in Section 5.1.

Re: M3 - We strengthen the theoretical arguments of UNI with a simple motivating example using a gaussian mixture model, presented in Section 4.3 of the revision (and the general response). We explain the necessity of unlearning and activation matching, by illustrating how a naive gradient descent approach (to minimise model confidence) omits more desirable baseline solutions that UNI is able to discover. We observe that UNI preserves path monotonicity, low-curvature and proximity and justify how it satisfies the weak dependence property for conformal path attribution.

Re: M4 - We thank the reviewer for their thought-provoking question. We discuss the broader contributions of UNI to XAI both in the general response and the revised paper manuscript. In particular, we are optimistic that UNI’s success will facilitate further research into 1) granular unlearning for hierarchical feature/concept attribution; 2) aligning XAI methods and models; 3) comparing network inductive biases and interpretability during training.

2. Minor comments
Re: m1 - While this work mainly focuses on white-box attribution methods, since black-box methods typically rely on different mechanisms (repeated forward passes instead of gradient information). We welcome the suggestion and have included a discussion of black-box methods in Section 6.

Re: m2 - Fixed and thanks.

3. Questions
Re: Q1 - We have not experimentally considered low-dimensional subspace paths, though this could potentially further improve the efficiency of UNI path attribution.

Re: Q2 - Under the reviewer’s suggestion, we expand UNI experiments to the NLP domain and report promising results on activation patching for interpreting language models (Appendix A.2).

Re: Q3 - This is an intriguing question which ties in with the UNI unlearning mechanism. We elaborate on this in both the revised manuscript and the general response:

UNI shows that first-order, sample-wise unlearning identifies and removes salient features predictive for a single sample. An exciting next step involves unlearning at the distribution level to localise and erase higher level concepts, such as unlearning a set of feature-clustered exemplars to erase common concepts. As noted by Reviewer Wrfy, unlearning in conjunction with activation matching can attribute features hierarchically, from local image textures, to intermediate shape prototypes, to class-semantic concepts.

We again express our gratitude to the reviewer for helping improve our work and look forward to a fruitful discussion!

Dear Authors,

Thank you for your detailed response.

I really appreciated the additional metrics which I think strengthens the paper. The Gaussian Mixture Model example adds much-needed theoretical clarity, excellent idea. Your discussion of UNI’s broader impact highlights its potential to inspire future work.

I’ve increased my score from 6 to 8 after reviewing your revisions. Good luck with the acceptance!

We thank the reviewer for their insightful feedback and thorough understanding of our work.

Re: W1 - We strengthen the theoretical arguments of UNI with a simple motivating example using a gaussian mixture model, presented in Section 4.3 of the revision (and the general response). We explain the necessity of unlearning and activation matching, by illustrating how a naive gradient descent approach (to minimise model confidence) omits more desirable and optimal baseline solutions that UNI is able to discover. We observe that UNI preserves path monotonicity, low-curvature and proximity and justify how it satisfies the weak dependence property for conformal path attribution.

Re: W2 - Under the reviewer’s recommendation, we expand UNI experiments to the NLP domain and report promising results on activation patching for interpreting language models (Appendix A.2). Equation 6 shows how UNI’s underlying principle of an unlearned point-of-reference can be adapted for an activation patching baseline. We report interpretability results on counterfactual prompts – we observe that like in the image domain, UNI improves the faithfulness of attributions for the 3 examined generative text models (Pythia-1b-v0, GPT2-medium, Llama-3.2-1B).

Re: W3 - To supplement the hyperparameter details given in Section 5, we discuss how crucial hyperparameters for explainability metrics (MuFidelity, deletion and insertion scores) are chosen for fair comparison in Section 5.1. Indeed, certain hyperparameters (e.g. |S| in MuFidelity score) have considerable impact on the signal-to-noise ratio of attributions, we provide justifications of hyperparameter settings and references to existing norms.

Re: W4 - This is an astute point. We incorporate discussions on how considerations of computational cost have influenced design choices of the UNI algorithm. Notably, UNI relies on a first-order method for unlearning, even though higher-order methods which leverage Hessian information should give better optimality bounds. However, computing the Hessian is expensive and error contributions of higher order terms quickly become negligible, which yields diminishing returns for high-order unlearning methods. Other design choices which impact performance include using only 1 unlearning step (with a large enough unlearning step size $\eta$), using fewer PGD activation matching steps $T$ for a sufficiently large PGD step size. Finally, since UNI computes stable, monotonic and conformal path features, it requires fewer Riemann approximation steps for path attribution, which drastically improves efficiency.

We again express our gratitude to the reviewer for their encouragement and help in improving our work. We look forward to a fruitful discussion!

We thank the reviewer for their feedback and thoughtful comments.

Re: W1 - We provide an illustrative example of UNI’s behaviour on a gaussian mixture model in Section 4.3 to clarify design choices of the UNI algorithm.

Re: W2 - We demonstrate that UNI is applicable beyond path-based image attribution methods by providing supplementary results in NLP. In particular, we adapt UNI to interpret generative language models, improving the stability and fidelity of recent activation patching methods in Appendix A.2. We observe that UNI’s underlying principle of finding an unlearned point-of-comparison is well aligned with XAI tasks: the unlearning mechanism lends itself to performing interventions on salient features, while activation patching constructs data probes for counterfactual model behaviour. Motivated by reviewers’ comments, we further discuss exciting follow-up questions prompted by UNI and its broader contributions to XAI in Section 7.

Re: Q1 - While the saliency scores attributed to image features is dependent on the model’s decision function (and the input sample), we believe that in Figure 2’s example, the black swan could be identified as being more predictive of the class label because of the animal’s prototypical, swan-like shape and because it is closest to the foreground. For the case of the swimsuit (now Appendix Figure 7), a likely reason is that models have been known to disproportionately rely on object boundaries for classification. While it appears that UNI does attribute greater salience to the lower body of the swimsuit, it is able to also trace the outlines of the upper body and provide a higher fidelity attribution than other baselines.

Re: Q2 - We leverage KL divergence to match the learned model’s output probabilities to the unlearned model’s probabilities. The KL divergence is particularly applicable because it is directed and asymmetric, which is important for activation matching in baseline construction. We have considered other objectives, such as simply using gradient descent to minimise model confidence and taking the minimiser as the baseline. However, as illustrated in Figures 6a and 6b, gradient descent only partially discovers suboptimal baseline candidates whereas the UNI algorithm discovers non-confounded and stable baselines for attribution.

We again express our gratitude to the reviewer for their effort in improving our work and look forward to a fruitful discussion!