Towards Faithful XAI Evaluation via Generalization-Limited Backdoor Watermark

Dear Reviewer RMSQ, we sincerely thank you for your valuable time and comments. We are encouraged by your positive comments on insightful analyses, extensive experiments, good method, good contribution, and good writing. We hope the following responses could help clarify potential misunderstandings and alleviate your concerns.

Q1: The only weakness I consider is that the authors haven't provided enough empirical evidence demonstrating the superiority of their proposed method over traditional backdoor-based techniques. For instance, they don't experimentally address why the minimum bounding box approach might yield inaccurate results as they claim.

R1: Thank you for this constructive comment! We are deeply sorry that our previous submission failed to explain it more clearly.

• Why taking absolute values of gradients in all (instead of parts of) evaluated SRV methods is better: Given an image and a trained model, existing SRV methods need to obtain the 'influence value' (e.g., gradient) of each pixel to model's prediction of the image before generating its saliency map. The influence value can be positive or negative. Its positive value indicates that increasing its pixel value will have a positive effect on the prediction. In general, to obtain the saliency map, we usually need to take the absolute value of each influence value before SRV methods filter out the most critical pixel positions (with sufficiently high influence scores). Otherwise, pixels having significantly negative effects on the prediction will not be selected. However, the existing backdoor-based SRV evaluation [13] only took the absolute value for BP while keeping the original influence value for other SRV methods (i.e., GBP, GCAM, GGCAM, OCC, FA, LIME). This inconsistent setting will lead to unreliable results. Accordingly, our standardized method is better than the traditional one.
• Why selecting $M$ pixel locations with the maximum saliency value (instead of using pre-defined threshold) as significant regions is better: The same threshold value may have significantly different influences across samples, even for the same SRV method, especially on different datasets. Accordingly, the saliency areas generated by the traditional method are not stable and reliable. However, using our TOP-K selection method can mitigate this problem and therefore it is better than the traditional method.
• Why calculating the IOU based on the significant regions directly is better than using minimum bounding box: Using the minimum bounding box may lead to a false sense of high performance. For example, consider a two-pixel trigger pattern where one pixel is located in the upper left corner and the other in the lower right corner. Assume that the saliency map is one pixel in the lower left corner and one pixel in the upper right corner. In this case, if the minimum bounding box is used, then IoU=1. However, this saliency map is completely wrong since there is no overlapping region. In other words, using traditional methods may result in IoU values that cannot correctly reflect the substantive effects of XAI methods. In contrast, using our method can avoid this problem. Accordingly, it is better than the traditional method.
• To further alleviate your concerns, we have also compared the results of the traditional method and those of our method. As shown in the following Table 1, the vanilla (i.e., traditional) method has various evaluations of the performance of the same SRV method on different datasets (especially for BP). In contrast, our standardized evaluation method has consistent rankings. These results verify that our method is more faithful.

Table 1. The average rank (based on the IOU) of SRV methods that is evaluated with the vanilla backdoor-based method and its standardized version on CIFAR-10 and GTSRB datasets

Please allow us to thank you again for reviewing our paper and the valuable feedback, and in particular for recognizing the strengths of our paper in terms of insightful analyses, extensive experiments, good method, good contribution, and good writing.

Kindly let us know if our response and the new experiments have properly addressed your concerns. We are more than happy to answer any additional questions during the post-rebuttal period. Your feedback will be greatly appreciated.

We would like to thank the reviewer for the helpful discussion during the first round of the review. We hope our response has adequately addressed your concerns. We take this as a great opportunity to improve our work and shall be grateful for any additional feedback you could give to us.

Dear Reviewer RMSQ,

We greatly appreciate your initial comments. We totally understand that you may be extremely busy at this time. But we still hope that you could have a quick look at our responses to your concerns. We appreciate any feedback you could give to us. We also hope that you could kindly update the rating if your questions have been addressed. We are also happy to answer any additional questions before the rebuttal ends.

Best Regards,

Paper3310 Authors

Dear Reviewer ZYFJ, we sincerely thank you for your valuable time and comments. We are encouraged by your positive comments on our meaningful and valuable exploration, non-trivial findings, novel and reasonable methods, comprehensive experiments, and good paper writing. We hope the following responses could help clarify potential misunderstandings and alleviate your concerns.

Q1: It would be better if the authors can provide more details about the differences between GLBW and BWTP. They seem similar in terms of formulas alone.

R1: Thanks for the insightful comment! We are deeply sorry that our submission may lead you to some misunderstandings that we want to clarify.

• Eq.(2) is different from Eq.(1). Formally, penalty loss (contained in Eq.(1)) and generalization loss (contained in Eq.(2)) have fundamental differences. Firstly, penalty loss is a maximization while generalization loss is essentially a minimization since its objective is $-\mathcal{L}$ instead of $\mathcal{L}$; Secondly, the constraint in penalty loss is $|m'|\leq t$ while that in generalization loss is $|m' \cap m| \leq \tau \cdot |m|$; Thirdly, there is a term $\mu \cdot |m'|$ in generalization loss that is not included in penalty loss.
• Penalty loss and generalization loss have fundamentally different meanings. In general, penalty loss synthesizes the potential trigger pattern and penalizes its effects based on its distance to the original one. If the potential trigger pattern is close to the ground-truth one, the predictions of watermarked DNNs to samples containing this pattern should be similar to the target label; Otherwise, their predictions should be similar to their ground-truth label. In contrast, generalization loss generates the most effective potential trigger pattern (other than the original one) and then minimizes its effects.
• Eq.(2) is feasible to reduce trigger generalization since it can generate potential trigger pattern (other than the original one) and minimizes its effects. In particular, as we mentioned in Section 4.3 and illustrated in Appendix (Section 4), we designed an adaptive optimization method to find promising synthesized patterns with the highest attack effectiveness and differences from the original trigger. It is also important for the success of our GLBW method.

We have added more details in Section 4.2 and Section 4.3 to better clarify Eq.(1) & Eq.(2) in our revision to avoid potential misunderstandings.

Dear Reviewer 4nxh, we sincerely thank you for your valuable time and comments. We are encouraged by your positive comments on our reasonable and insightful method, solid experiments, good contribution, and good writing. We hope the following responses could help clarify potential misunderstandings and alleviate your concerns.

Q1: Actually, In my opinion, the trigger in backdoor attack and the real object are different in essence. In an image, the trigger is an out-of-context object, and the real object is in context. Therefore, although a SRV evaluation method can show good performance on the trigger, it does not ensure the effectiveness on the real object. I wonder that the authors whether provide some analysis about this difference.

R1: Thank you for your insightful question! We admit that our GLBW and other existing backdoor-based SRV methods all rank SRV methods based on their performance in explaining simple backdoored patterns (instead of high-level complicated real object features). However, it not necessarily means that our method is impractical. We hereby provide more explanations:

• In practice, humans categorize a given image based on its local regions in most cases. For example, we only need to see the image areas of its 'head' to know it is a bird. These local regions are similar to the trigger patterns (e.g., a patch) used in our method. Accordingly, results made by our method can be a good reference in real images for practice.
• Currently, it is impossible to faithfully evaluate the performance of SRV methods for complicated features since there is no ground-truth salience map for them. Even a human expert cannot mark the salience map for complicated features.
• Simple backdoored patterns are also features used by DNNs for their predictions. Accordingly, the evaluation of SRV methods on trigger features is the first and the most important step toward evaluating their general performance and is therefore of great significance.

We have provided more details and explanations in Appendix C (i.e., Why Do We (Only) Need to Consider Patch-based Triggers?) of our revision.

Q2: In the experiments, although they conduct many experiments, it seems that there are not some comparisons with the SOTA SRV evaluation methods. In Section 2.2, the authors discuss many existing SRV evaluation methods, they should give the comprehensive comparisons with these methods.

R2: Thank you for this constructive comment! We are deeply sorry that our previous submission may lead you to some misunderstandings that we want to clarify here.

• Currently, the most reliable evaluation of SRV methods is still based on the human inspection. However, human evaluation is costly and time-consuming. As such, in practice, we need to design automated assessment methods like our GLBW-based one. Specifically, we have compared human evaluation as a reference to our method in Appendix K to some extent. As shown in the following table 1, the rankings generated by our GLBW method are similar to those made by people, verifying the effectiveness of our GLBW.

Table 2. The average rank of SRV methods that is evaluated with our GLBW method and human inspection on the GTSRB dataset.

• For automated evaluation methods, except for the vanilla backdoor-based method compared in our paper, all other methods have been shown to consume significant computational resources and have limited accuracy (Hooker el al, 2019). Accordingly, we did not compared our methods with them.
• Different from classical tasks such as image categorization, the evaluation of XAI methods is very difficult because there is no ground-truth saliency areas. However, backdoor-based methods, especially our GLBW-based one, could lead reliable saliency map evaluation since the ground-truth saliency areas are those of the trigger pattern when there is no trigger generalization. This is why backdoor-based assessment methods are well worth further research.

Please allow us to thank you again for reviewing our paper and the valuable feedback, and in particular for recognizing the strengths of our paper in terms of reasonable and insightful method, solid experiments, good contribution, and good writing.

Kindly let us know if our response and the new experiments have properly addressed your concerns. We are more than happy to answer any additional questions during the post-rebuttal period. Your feedback will be greatly appreciated.

We would like to thank the reviewer for the helpful discussion during the first round of the review. We hope our response has adequately addressed your concerns. We take this as a great opportunity to improve our work and shall be grateful for any additional feedback you could give to us.

Dear Reviewer 4nxh,

We greatly appreciate your initial comments. We totally understand that you may be extremely busy at this time. But we still hope that you could have a quick look at our responses to your concerns. We appreciate any feedback you could give to us. We also hope that you could kindly update the rating if your questions have been addressed. We are also happy to answer any additional questions before the rebuttal ends.

Best Regards,

Paper3310 Authors

Dear Reviewer LeTy, we sincerely thank you for your valuable time and comments. We hope the following responses can help clarify potential misunderstandings and alleviate your concerns.

Q1: Unnecessity of the consideration of trigger generalization in evaluating XAI. It is unclear why the potential triggers should be considered in evaluating XAI based on watermark regions. For example, assume that there is a cat image and a poisoned model predicts it as a cat. If the prediction results changed by adding a watermark on the image, then it is due to the watermark in a high probability.

R1: Thank you for this insightful quesion! We are deeply sorry that we failed to explain it more clearly in our submission. In general, position generalization will make the result of backdoor-based SRV evaluation less reliable. More details are as follows.

• Backdoor-based SRV evaluation methods rely on a latent assumption that existing backdoor attacks have no trigger generalization, i.e., only the trigger used for training (dubbed 'original trigger') can activate backdoors. These methods believe that trigger regions should be treated as the regions that contribute the most to the model's prediction (i.e., target label) of poisoned samples because their ground-truth labels are not the target label. Accordingly, they use the area of original trigger as the ground-truth reference and calculate the average intersection over union (IOU) between it and the saliency areas generated by the SRV method of the backdoored model over different backdoored samples as an indicator to evaluate the SRV method.
• This assumption does not hold when backdoor watermark has position generalization. As we shown in our Figure 4, there are many potential trigger patterns other than the original one that can still activate backdoors. In other words, this assumption does not hold for existing backdoor watermarks.
• Its failure may lead to unreliable results. For example, given a SRV method, assume that its generated saliency areas of most poisoned samples are only a small part of that of the original trigger. According to the existing backdoor-based evaluation, this SRV method will be treated as having poor performance since it has a small IOU. However, due to the generalization of backdoor watermarks, the model may only learn this local region rather than the whole trigger. In this case, the evaluated SRV method is in fact highly effective, contradicting to the results of the backdoor-based SRV evaluation.

We have added more details in the introduction of our revision to make it more clearly.

Q2: Limitation of IoU based evaluation methods. Since the saliency map is not a segmentation, comparing the ground truth segmentation label with the saliency map could not be the best evaluation method. The classifier would not uniformly refer to the ground truth area.

R2: Thank you for this insightful comment!

• We admit that classifiers may not uniformly refer to the ground truth area. However, it only leads to the fact that we can hardly obtain the ground-truth saliency areas with their weights. Arguably, this is not the reason why our IoU-based method is limited, as all evaluation methods are unable to obtain such ground-truth weights.
• We never claim that IoU-based evaluation is the best method. We admit that it may not be the best, but at least it fits due to the similarities of this task and segmetation.
• We adopt the IoU-based method simply following the setting of our baseline (Lin et al., 2021) for a fair comparison.
• However, we fully understand your concern about whether using other metrics may change the results. To further alleviate your concerns, we conduct additional experiments on the variant of our standardized method (dubbed 'standardized-L2'), where we adopt $L_2$ distance instead of IoU for the evaluation. As shown in the following table, our standardized-L2 can still reaches consistent rankings. In contrast, the vanilla method has various evaluations of the performance of the same SRV method on different datasets (especially for BP). These results verify that our evaluation is more faithful and scalable to evaluation metrics.

Table 1. The average rank of SRV methods that is evaluated with the vanilla backdoor-based method and its standardized versions on CIFAR-10 and GTSRB datasets.

Please allow us to thank you again for reviewing our paper and the valuable feedback. These comments are valuable for us to improve our paper further.

Please kindly let us know if our response and the new experiments have properly addressed your concerns. We are more than happy to answer any additional questions during the post-rebuttal period. Your feedback will be greatly appreciated.

We would like to thank the reviewer for the helpful discussion during the first round of the review. We hope our response has adequately addressed your concerns. We take this as a great opportunity to improve our work and shall be grateful for any additional feedback you could give to us.

Dear Reviewer LeTy,

We greatly appreciate your initial comments. We totally understand that you may be extremely busy at this time. But we still hope that you could have a quick look at our responses to your concerns. We appreciate any feedback you could give to us. We also hope that you could kindly update the rating if your questions have been addressed. We are also happy to answer any additional questions before the rebuttal ends.

Best Regards,

Paper3310 Authors

Thank you for providing a thorough rebuttal to the reviewer's comments. I appreciate the effort you've put into addressing the concerns. Also, I appreciate the use of red text for revised manuscripts.

Many of my concerns are resolved, but I still respectfully disagree with your notions regarding R1. The main argument is whether the existence of potential triggers $x_{adv}$ truly matters for backdoor-based evaluation. To discuss more, I would denote the benign input, poisoned image, and the potential triggers as $x$, $x_p = G(x)$, and $x_{adv}$, respectively.

• Toy example

Consider a specified scenario where the input is denoted as $x\in\mathbb{R}^{d+3}$. In this context, the first $d$ dimensions determine the true label, while the last $3$ dimensions consistently hold zero value according to the original distribution. An adversary trains a poisoned model by manipulating the last three dimensions: the model generates opposite predictions when any of the last three elements is set to 1. Specifically, $G(x; 0) = (x_1, x_2, ..., x_d, 1, 0, 0)$, $G(x; 1) = (x_1, x_2, ..., x_d, 0, 1, 0)$, and $G(x; 2) = (x_1, x_2, ..., x_d, 0, 0, 1)$. For the purpose of backdoor-based XAI evaluation, only $G(x; 0)$ is utilized as a poisoned image. Consequently, $G(x; 1)$ and $G(x; 2)$ remains as potential triggers.

In this case, my opinion is that the existence of potential triggers $G(x; 1)$ and $G(x; 2)$ could not affect the prediction results of $G(x; 0)$ and so do for the backdoor-based XAI evaluation. The prediction will totally be made by mostly referring to $x_{d+1}$, and the XAI should highlight $x_{d+1}$ as well, regardless of the existence of potential triggers.

• Random location trigger

Consider a poisoned model trained to be activated on the random location trigger, as in Figure 4 (b, d). Obviously, these models will always have potential triggers. The assumption of this paper is that these models cannot be used for a backdoor-based XAI evaluation, regardless of the reliability of this model.

In summary, the main motivation of this paper is from the assumption:

Backdoor-based SRV evaluation methods rely on a latent assumption that existing backdoor attacks have no trigger generalization.

I maintain my original stance; I cannot fully endorse the assumption mentioned. This assumption plays a pivotal role throughout the paper, and its validity is of utmost importance. I am inclined to keep my initial score, but I am open to reconsidering and potentially increasing it if the supplement addressing this particular assumption is provided. A more comprehensive discussion or additional supporting evidence on this matter would significantly contribute to strengthening the overall credibility of the paper.