DAME: A Distillation Based Approach For Model-agnostic Local Explainability

We thank the reviewer 7xDx for reviewing the draft and providing valuable comments.

1. clear, well contextualized: Thank you for the appreciating our work.

2. subjective evaluation not informative: Thank you for the pointer on the evaluation.

• We agree that there is no single way to explain away the decisions made by models. Hence, we evaluate in a multi-pronged way, using fidelity measure, IoU, and the human subjective study.
• The motivation for the human evaluation is that, for the samples in which the black-box model is highly confident, how do the humans evaluate saliency maps generated by different XAI methods under the assumption that the black-box model, on these samples with high classification confidence, has attended to the most salient parts of the objects.
• There are prominent studies and regulations that highlight the importance of explanations being human comprehensible as they are the end users [2,3,4,5].
• The collective evaluation with multiple metrics highlighted the advantage of DAME.

5. DNN not use human-like features .. penalized by IoU: The IoU based evaluation is based on the assumption that, the classifier relies on the target object region for condidently right classified samples. To validate:

• We mask away the regions of the target object annotations, and these masked images leads black box accuracy going down from $92.3$% to $18.3$%.
• However, masking away the regions outside the object class and keeping the object annotations leads to $76.5$% accuracy. Although a drop from $92.3$%, it validates that the annotations as primary information used by ViT model to classify the objects.
• As pointed by the reviewer, the misclassification may be due to the short-cut learning. To probe this, we computed the IoU on the mis-classified samples while explaining them using ground truth labels and obtained IoU of $16.3$% ($31.4$% on correct samples) using DAME. It indicates that the model has failed to focus on the object regions. The drop in IoU for other XAI methods are:

6. Deletion measure: We clarify the evaluation carried in the draft are based on removing $30$% pixels at once.

Reason behind $30$% feature deletion: For images ($x^{(i)}$) and object annotations ($Z^{(i)}$), as shown in Figure 20, let the object occupancy is:

$

\mu_O^{(i)} = \frac{|(m,n): Z^{(i)}(m, n)=1, (m,n)\in Z^{(i)}|}{|(m,n): (m,n)\in Z^{(i)}|}X 100%

$

The mean object occupancy $\mu_O = \frac{1}{|D|}\sum_{k=1}^{|D|}\mu_O^{(i)}$ is found to be $29.2$% ($D$ denotes the VOC dataset) as shown in Figure 20 and hence $30$% important feature removal is used as a fast, one-shot approach. If gradual deletion is performed in $x$ steps, it will require $10x$ inference time.

However, we also did gradual deletion as a standard way and found the (AUC- {LIME: 0.340, RISE: 0.332, DAME: 0.325}) as shown in Figure 19. [1] reported only drop in model confidence but we reported drop in accuracy also. Also note the plots and AUC values in [1] are for selected samples, but we have reported the metrics over full VOC dataset; and DAME shows the most sensitivity to deletion.

7. proposed framework with linear models is missing: We used a linear student network in DAME and observe that the model fails to generate meaningful explanations. The fidelity measure at $30$% feature deletion results in only $24.3$% drop in accuracy as opposed to $56.2$% for the non-linear model.
8. similar decisions for similar reasons?: We removed the top-k features (based on XAI output) gradually from the image and pass the masked images through the DAME student model as well as the black-box model to get the scores. The DAME scores show high similarity with the black box scores with $R^2$ of $0.819$ on VOC dataset, confirming their agreement.

References:

1. Petsiuk, Rise: Randomized input sampling for explanation of black-box models.
2. Nguyen. “The effectiveness of feature attribution methods and its correlation with automatic evaluation scores”. NeurIPS 2021.
3. Colin, What i cannot predict, i do not understand: A human-centered evaluation framework for explainability methods. NeurIPS 2022.
4. Goodman. “European Union regulations on algorithmic decision-making and a “right to explanation””. AI magazine 2017.
5. Doshi. "Towards a rigorous science of interpretable machine learning." arXiv 2017.
6. Sun, Ao, "Explain Any Concept: Segment Anything Meets Concept-Based Explanation.", NeurIPS 2023.
7. Lerman, Samuel,. "Explaining Local, Global, And Higher-Order Interactions In Deep Learning." ICCV 2021.

Thank you for addressing some of my points, I will raise my score accordingly.

2. I still stand by my original argument.
3. I appreciate the validation done for the IoU measure, but I still do not see its value in evaluating methods when you already use the standard fidelity metrics, especially if a fidelity-like metric is needed to validate it. Again, we are not sure that the model only cares about human-like features and we are not sure that the XAI method highlights the feature used by the model, yet using IoU requires us to make assumptions about either or both of them, hence I would be less confident in the results.
4. Thank you for the clarification, the motivation for this choice is sound.
5. and 8. Thank you for addressing my questions.

We thank the reviewer hRgW for reviewing the draft and providing valuable input.

1. addresses an important question on generating saliency map-based explanations with only black-box access:

Thank you for appreciating the motivation of our work.

2. Besides traditional tasks from Computer Vision, the paper also reports results on audio processing tasks.:

Thank you for appreciating our experimental setup on diverse tasks.

3. The paper is well-written and easy to follow.:

Thank you.

4. Even though they consider a diverse set of tasks, the authors must add additional experiments to strengthen the paper. It is hard to see whether the proposed framework offers a clear advantage over the baselines (as explanations are typically subjective).:

We have performed a set of additional experiments to further highlight the advantages of DAME over other methods, and to experimentally validate the choices made in the paper.

• Score agreement with black box: We carried out experiments to examine how much the model based XAI methods (LIME and DAME) agree with the black box behavior for perturbed samples- commonly used as one of the fidelity measures. LIME and DAME tries to mimic the black box at the local neighbourhood and hence their response ideally should match with black box response for samples drawn from local neighbourhood of the input. We observe a substantially stronger agreement for DAME over LIME, as shown in Figure 22 in the revised draft.
• Gradual deletion based fidelity evaluation: We performed gradual deletion experiment (as proposed in [1]) instead of removing $30$% of the top features at once (as reported in the original draft). We observed a lower value of area-under-the-curve (AUC) for DAME, demonstrating its improved sensitivity to deletion, as shown in Figure 19 in the revised draft.
• Probing wrongly classified samples: The XAI for object/event classification should be able to shed some light on mis-classifications by the black box, apart from explaining the correctly predicted samples. To examine this, we explored generating saliency based explanations for the mis-classified samples in VOC dataset and compared them with the ground truth annotations. We found that the mean IoU, using different XAI methods, elicit a drop for all XAI methods, with the largest drop for DAME approach, as shown in Table 2.1 below. It indicates that the black box model may have focused on the contextual regions (outside the target class bounding box) for these samples, leading to mis-classification. | method | mean IoU | IoU drop from correct samples | |------------|----------------|----------------------------------------------| | RISE | 19.7 (14.2) | 11.4 | | LIME | 16.6 (13.2) | 10.2 | | DAME | 16.2 (13.9) | 15.2 | Table 2.1: IoU for misclassified samples.
• Justification of map generation network: Additional experiments regarding the need of the map generation network is performed, which justifies the two network strategy in the DAME framework as discussed in Table 2.3 below.
• Additional ablation experiments: Architecture choice of student network, and the minimum number of perturbation samples needed for the DAME framework are experimentally established as discussed in Tables 2.2 and 2.4 below.

5. Does the architecture of the map generation & student network affect the performance? Does it need to be shallow or deeper? What are the design considerations for these networks?:

We experimented with different architectures of the student network. Using a two layer fully connected network (FCN) following a 2 layer convolutional network (CNN) as student network gives significantly better performance than a other architecture chices as shown in Table 2.2 below.

Regarding depth, a student network which is deeper, increases the time overhead to generate the explanations for each input sample. As we attempt a local approximation only, a shallow network may be sufficient which also provides significant savings in computation time.

References:

1. Petsiuk et al. RISE: Randomized input sampling for explanation of black-box models. BMVC. 2018

We thank Reviewer hRgW for his valuable time in going through our response and appreciating, and also increasing the score. We will surely include the responses in the final version of the draft.

We thank the reviewer RyMT for reviewing the draft and providing valuable input.

1. works in input space and not in the binary mask space, establishes the shortcoming of locally linear approximations a small toy experiment, results from many varied experiments with both quantitative metrics and qualitative samples, intuitive approach, well written and easy to follow, appendix is very thorough and quite informative:

Thank you for appreciating our problem formulation, presentation and evaluation on diverse set of tasks and datasets.

2. The biggest bottleneck to using this approach would be having to train a whole new model to understand the behavior of the model for one single input sample.:

Thank you for pointing this out. Because of unavailability of gradient access to black box in gradient-free explainability, all post-hoc methods (LIME, DAME, and RISE) rely on generating local perturbations from the input and approximating the corresponding black box responses. In this setting, DAME takes about $50$ sec. of computation time (single A-6000 NVidia GPU on an Intel server) to train on a single sample. The reference numbers for LIME: 25 sec., and RISE: 40 sec. for the same settings. Hence, DAME is only about $25$% more expensive compared to baseline method of RISE, while yielding performance gains on various tasks and evaluation metrics listed in the paper and the responses.

3. Results from RISE are often quite competitive in tables 1 and 2. Smooth Grad is also quite competitive.:

SmoothGrad requires internal access to the model in the form of gradients to generate explanations, which is highly restrictive for various black-box models. The proposed approach is gradient-free post-hoc XAI setting, with only input-output access (as described in the paper, various recent models like GPT are released without any internal access to the model architecture). Hence, gradient based XAI methods merely correspond to upper-bound for gradient free methods. Further, as seen in Table I of the main draft, gradient methods like SmoothGrad perform poorly on Transformer based models like ViT.

With regard to comparison with RISE, the proposed DAME framework,

• Improves in IoU metric over the RISE on all the audio based evaluations.
• Improves over the RISE/LIME on image based evaluations using the fidelity based measures, for example, deletion (please refer to Section A.12 and Figure 19 in revised draft).

Also The XAI for object/event classification should be able to shed some light on mis-classifications by the black box, apart from explaining the correctly predicted samples. To examine this, we explored generating saliency based explanations for the mis-classified samples in VOC dataset and compared them with the ground truth annotations. We found that the mean IoU, using different XAI methods, elicit a drop for all XAI methods, with the largest drop for DAME approach as shown in below Table. It indicates that the black box model may have focused on the contextual regions (outside the target class bounding box) for these samples.

• Subjective tests indicated improved preference for the explanations provided by DAME over the other methods.

4. This approach is akin to a gradient-based approach in the guise of gradient-free. If one was to distill the whole black-box into another model (not just in the sample’s neighborhood) and then apply any gradient-based method, I believe that that would be much simpler. why the authors have taken this round-about route.:

Thank you for raising it, which also warrants a discussion in the main draft about the motivation for DAME. We argue why the global distillation is not a feasible choice for post-hoc XAI methods.

• Global distillation requires access to the training data of the original teacher (black-box) model. As DAME is proposed for a post-hoc XAI setting, the requirement of access to the supervised training data might pose a severe limitation to the applicability of the XAI (for example, an XAI for an LLM might need a very training corpus to generate a suitable student model).
• Simulating the classification bias and spurious correlation behavior of the larger black-box with a simpler student model globally might be infeasible, which may cause a divergent explanation using the student model compared to the explanations generated for the teacher model.

We thank the reviewer kHxm for reviewing the draft and providing valuable inputs.

1. The proposed method is comprehensively evaluated on image and audio datasets using a number of evaluation techniques and shows improvement over existing local explainability methods.: Thank you for appreciating our work and the evaluations performed on on diverse datasets and tasks.

2. it makes sense to use a non-linear approach, The approach proposed to generate the local saliency explanation is novel, The evaluation is quite comprehensive including fidelity-based, subjective and qualitative evaluations and comparison with 9 XAI methods: Thank you appreciating our approach, novelty and evaluations.

3. Based on the 3 loss functions that need to be handled, it seems likely that the method may not work out of the box: Thank you for the question. Although it is true that the loss function has three components, we found that tuning them for a modality works well for multiple applications with that modality. For example, the hyperparameters tuned for ImageNet was used as is for VOC dataset, and hyperparameters tuned for ESC-10 based audio classification were used as is for the COSWARA application.

Moreover, we would also like to add that the KL-div loss component contributes minimally as can be seen from the Table below. Hence, the MSE loss along with a L1 loss component (to avoid singularity) itself may constitute a reasonable choice for the loss function in DAME.

4. How is local vicinity and distance between the given sample and perturbations in DAME: We used the same measure of distance between a sample and its perturbations, the L2 distance between them in the input space. On the contrary, LIME uses the distance as the L2 distance between them in the binary space ($M_{x^{(k)}}$ space, as described in Section 4.2 in the draft). The distance measure in input space results in higher sensitivity because masking-off larger regions and smaller segments may not be similar.

5. In case of DAME, can the authors comment on local invariance of explanations (do similar inputs yield similar explanations)?: Yes, we have verified this experimentally that similar inputs indeed generate similar explanations. We have also performed an attribution experiment between the student and teacher model. We removed the top-k features (based on XAI output) gradually (k increasing from 0 to 100%) from the image and pass the masked images through the DAME student model as well as the blackbox model to get the scores. The DAME scores show high similarity with the black box scores. An average coefficient of determination ($R^2$) of $0.819$ was obtained for the samples in VOC dataset, confirming the agreement between DAME student and the black box model.

6. Would DAME be impacted by correlated featured?: Examining sensitivity of XAI methods with respect to correlated features is an important area of research. While a detailed quantitative evaluation was not possible within the short rebuttal period, we have resorted to a qualitative evaluation. An example image of a cat with correlated features (the shadow associated with it) and the explanation generated by DAME is shown in Figure 21 of the updated draft.
   The shadow has similar shape as the cat, leading to a correlated feature in the input. However, the DAME generated explanation is not influenced by the presence of the shadow.

7. Instead of using a masking approach to generate perturbations (e.g. LIME), if we have a realistic distribution of perturbed images (e.g. MeLIME), can the DAME pipeline still be used?: We thank the reviewer for pointing us to this resource. The proposed approach is indeed agnostic to the choice of masking or perturbation generation approach. MeLIME proposes an effective perturbation strategy in general, that improves the explanations provided by different methods like LIME. Because this perturbation strategy essentially focuses on generating samples that better capture the representation of local vicinity of the input, it constitutes a promising experiment to explore before the camera ready version of the submission.