Causality-Based Black-Box Backdoor Detection

Q1. More explanations on the choice of magnitude set.

Thanks for the valuable suggestion. We will answer the question with the following aspects:

• [Already Discussed: The tradeoff behind magnitude set] Firstly, we have already briefly discussed the choice of magnitude set in the Appendix. To sum up, the choice of magnitude set is a tradeoff between efficiency and effectiveness. If the magnitude set is too short, then the granularity might not be fine-grained enough to distinguish between backdoor samples and clean samples, which is detrimental to meeting the effectiveness requirement. However, if the set is too large, the efficiency requirement cannot be satisfied.

• [Yet Discussed: How the parameters impact the choice of magnitude set] To further clarify the confusion, we conduct sensitivity testing on the choice of magnitude set with the following experiment on the Cifar10, GTSRB, and Imagenet-subset. We formalize a magnitude set with two parameters: step_length and magnitude_length, where step_length is the gap between two noise magnitudes and magnitude_length is the number of noise magnitudes.

We conduct sensitivity testing on the choice of magnitude set against four backdoor attack methods over three datasets. A quick view of the result is: https://ibb.co/m0m97Lm, where the x-axis (horizontal) denotes the step_length, the y-axis (vertical) denotes the length of the magnitude set, and the value in each cell denotes the AUC score. It is seen that the method performs well when we choose the magnitude set with step_length=0.2 and length=7. These results have been added to the Appendix G of the updated manuscript.

Q2. More experiments on $\alpha, \beta$.

Thanks for the insightful comment. In our setting, $\alpha$ and $\beta$ are not hyper-parameters that need to be tuned but some fixed parameters that can be pre-defined. Specifically, $\alpha$ value is fixed as 1 and $\beta$ value is fixed as $|S|-1$. We will answer the reasoning behind the choice with the following two aspects:

Firstly, the choice is highly driven by our causality analysis in Section 2. Specifically, the sample-specific backdoor samples will show an abrupt prediction flipping if a small amount of noise is added, while sample-agnostic backdoor samples will show a consistent prediction until a considerably large noise is added. Therefore, $\alpha=1$ value is set to capture the sample-specific backdoor samples, and $\beta=|S|-1$ value is set to capture the sample-agnostic backdoor samples. We argue that setting $\alpha$ and $\beta$ to other values may lead to misalignment with the causality analysis.

Secondly, the provided experiments across Section 4 all demonstrate that this choice is able to perform well on all three datasets against four types of backdoor attacks.

To sum up, we can readily use fixed value $\alpha = 1$ and $\beta= |S|-1$ when applying our backdoor detection method in practice.

Q3. More explanations on Table 1. Thanks for the contributive suggestions. In Table 1, BadNet and Blend correspond to the sample-agnostic backdoor attacks since their trigger patterns are universal, while WaNet and ISSBA correspond to the sample-specific backdoor attacks since their trigger patterns are customized for each individual sample.

Therefore, the experimental results in Table 1 already validate the effectiveness of our method on both types of backdoor attacks.

Q4. Clairifcations on Figure 2(f).

Thanks for the comments. We totally agree with your point that the prediction might not be "fish". We have accordingly revised the figure in the updated manuscript.

Dear Reviewer,

Thanks for your assessment and your encouragement of our work. As the deadline for reviewer-author discussion (11/22/23) approaches, we would be happy to take this opportunity to make more discussions about any of our questions or concerns. If our response has addressed your concerns, we would be grateful if you could re-evaluate our paper based on our feedback and new supplementary results.

Sincerely,

Authors

Dear Reviewer,

Thanks for your assessment and your encouragement of our work. As the deadline for reviewer-author discussion (11/22/23) approaches, we would be happy to take this opportunity to make more discussions about any of our questions or concerns. If our response has addressed your concerns, we would be grateful if you could re-evaluate our paper based on our feedback and new supplementary results.

Sincerely,

Authors

Q3. More explanations on how to choose $\alpha$ and $\beta$.

Thanks for the insightful comment. In our setting, $\alpha$ and $\beta$ are not hyper-parameters that need to be tuned but some fixed parameters that can be pre-defined. Specifically, $\alpha$ value is fixed as 1 and $\beta$ value is fixed as $|S|-1$. We will answer the reasoning behind the choice with the following two aspects:

Firstly, the choice is highly driven by our causality analysis in Section 2. Specifically, the sample-specific backdoor samples will show an abrupt prediction flipping if a small amount of noise is added, while sample-agnostic backdoor samples will show a consistent prediction until a considerably large noise is added. Therefore, $\alpha=1$ value is set to capture the sample-specific backdoor samples, and $\beta=|S|-1$ value is set to capture the sample-agnostic backdoor samples. We argue that setting $\alpha$ and $\beta$ to other values may lead to misalignment with the causality analysis.

Secondly, the provided experiments across Section 4 all demonstrate that this choice is able to perform well on all three datasets against four types of backdoor attacks.

To sum up, we can readily use fixed value $\alpha = 1$ and $\beta= |S|-1$ when applying our backdoor detection method in practice.

Despite that $\alpha$ and $\beta$ are fixed values, we can tune the choice of magnitude set $|S|$ to achieve a better performance. Specifically, there are two hyper-parameters corresponding to the choice of magnitude set, i.e., step length and magnitude length. We conduct extensive experiments against all backdoor attack methods over three datasets to evaluate the performance of our method with different choices of step length and magnitude length. A quick view of the result is https://ibb.co/m0m97Lm. where the x-axis (horizontal) denotes the step_length, the y-axis (vertical) denotes the length of the magnitude set, and the value in each cell denotes the AUC score. It is seen that the method performs well when we choose the magnitude set with step_length=0.2 and length=7. These results have been added to the Appendix G of the updated manuscript.

Q4. Sensitive to Different Network Architectures

Really thanks for the insightful comment. We first argue that our method is model-agnostic and architecture-agnostic. As the setting suggests, our method only depends on two sources of information: (1) input image and (2) prediction labels from DNNs. Therefore, we do not assume any prior information about the model architectures. Besides, we have actually also evaluated our method with different architectures in our experiment. For example, we adopt ResNet architecture for Cifar10 and GTSRB datasets, and EfficientNet architecture for ImageNet-subset dataset. All the experiments have already validated the effectiveness of our method. That being said, we still value your suggestion as this is a very good way to further demonstrate the applicability of our method. We will add more experiments in the updated manuscript.

[1] Turner, Alexander, Dimitris Tsipras, and Aleksander Madry. "Label-consistent backdoor attacks." arXiv preprint arXiv:1912.02771 (2019).

[2] Barni, Mauro, Kassem Kallas, and Benedetta Tondi. "A new backdoor attack in cnns by training set corruption without label poisoning." 2019 IEEE International Conference on Image Processing (ICIP). IEEE, 2019.

[3] Liu, Yunfei, et al. "Reflection backdoor: A natural backdoor attack on deep neural networks." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part X 16. Springer International Publishing, 2020.

Q1. Experiments on More Baseline Attack Methods.

Thanks for the valuable suggestions. We provide the following experimental results on the mentioned clean-label backdoor attack methods (Refool[3], SIG[2], and Label-clean[1]). Each cell in the following table contains three values, which are precision (P), recall (R), and AUROC, respectively.

As the table shows, our method generally has a significant advantage over the other baseline defense methods on the three metrics. We have added the experimental results in our updated manuscript. It is also noted that a full table of added baseline attack methods is provided in Table 1 at the Global Response.

Q2. Some obvious typos.

Thanks for your valuable comments. We have revised the typos in the updated manuscript.

Dear Reviewer,

Thanks for your assessment and your encouragement of our work. As the deadline for reviewer-author discussion (11/22/23) approaches, we would be happy to take this opportunity to make more discussions about any of our questions or concerns. If our response has addressed your concerns, we would be grateful if you could re-evaluate our paper based on our feedback and new supplementary results.

Sincerely,

Authors

Dear Reviewer,

Thanks for your assessment and your encouragement of our work. As the deadline for reviewer-author discussion (11/22/23) approaches, we would be happy to take this opportunity to make more discussions about any of our questions or concerns. If our response has addressed your concerns, we would be grateful if you could re-evaluate our paper based on our feedback and new supplementary results.

Sincerely,

Authors

Q1. Experiments with more baseline attack methods

Thanks for the valuable suggestions. We provide the following experimental results on the mentioned backdoor attack methods (TaCT[1] and Adaptive Blend[2]). Each cell in the following table contains three values, which are precision (P), recall (R), and AUROC, respectively.

As the table shows, our method generally has a significant advantage over the other baseline defense methods on the three metrics. We have added the experimental results in our updated manuscript. It is also noted that a full table of added baseline attack methods is provided in Table 1 at the Global Response.

Q2. Theoretical Rationale for Utilizing Counterfactual Samples.

Thanks for the insightful suggestion. Firstly, we argue that introducing counterfactual samples is not a randomly proposed strategy, but is highly supported by the theoretical causal analysis in Section 2 and highly motivated by a real-life example of identifying salt and sugar provided in Section 2.3. That being said, we agree that adding more mathematical justifications on why we utilize counterfactual samples could make the paper more theoretically sound. Therefore, we will work on this and provide our theoretical results in the updated manuscript.

[1] Tang et al., "Demon in the Variant: Statistical Analysis of DNNs for Robust Backdoor Contamination Detection."

[2] Qi et al., "Revisiting the Assumption of Latent Separability for Backdoor Defenses."

Dear Reviewer,

Thanks for your assessment and your encouragement of our work. As the deadline for reviewer-author discussion (11/22/23) approaches, we would be happy to take this opportunity to make more discussions about any of our questions or concerns. If our response has addressed your concerns, we would be grateful if you could re-evaluate our paper based on our feedback and new supplementary results.

Sincerely,

Authors

Dear Reviewer,

Thanks for your assessment and your encouragement of our work. As the deadline for reviewer-author discussion (11/22/23) approaches, we would be happy to take this opportunity to make more discussions about any of our questions or concerns. If our response has addressed your concerns, we would be grateful if you could re-evaluate our paper based on our feedback and new supplementary results.

Sincerely,

Authors

Q7. Experiments with more complicated BadNet patterns.

Thanks for the suggestions. In the default setting, we use grid-like $3\times3$ pixel squares as the trigger pattern(Figure 9). Moreover, we have also compared the performance of our method when the size of the BadNet trigger patterns varies (e.g., $2\times 2, 4\times 4, 5\times 5, ...)$ in Figure 7. To further demonstrate the effectiveness of our method against more complicated BadNet trigger patterns, we provide some additional results on BadNet when the trigger pattern is randomly generated by torch.rand(3,3).

Q8. Adaptive attacks analysis.

Thanks for the valuable suggestion. Suppose an attacker already knows our defense methods in advance, then intuitively the attacker will train the DNN model with the following adaptive training loss function:

$\min_{\theta} \sum_{i=1}^{|D_{/c}|} \ell (f_{\theta}(x_i ), y_i) + \sum_{i=1}^{|D_b|} \ell (f_{\theta}(x_i'), y_t) + \sum_{i=1}^{|D_b|} \ell (f_{\theta}(x_i' + m * \epsilon_i), y_i)$

where $\epsilon_i \sim N(0,1)$ is a random Gaussian noise added to each backdoor sample $x_i'$, and $m$ is the magnitude multiplier of the added noise. Intuitively, this loss function will make the prediction behaviors on backdoored images (when the noise is added) more similar to those on clean images. To evaluate whether our method is still effective against adaptive attacks, we provide results with varying $m$ against BadNet attack on the Cifar10 dataset in the following table.

It can be inferred that our method can still achieve a satisfactory performance when the noise magnitude is small (e.g. < 0.5), but the performance will gradually drop when the noise magnitude multiplier becomes higher (e.g., > 1.0). However, we argue that when the noise magnitude multiplier > 1.0, the whole image will be dominated by random noise, rendering the modified image easily detected and filtered out by the off-the-shelf backdoor filtering algorithms.

[1] Zhang, Z., Liu, Q., Wang, Z., Lu, Z., & Hu, Q. (2023). Backdoor Defense via Deconfounded Representation Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12228-12238).

[2] Cheng, S., Liu, Y., Ma, S., & Zhang, X. (2021, May). Deep feature space trojan attack of neural networks by controlled detoxification. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 2, pp. 1148-1156).

[3] Liu, Y., Lee, W. C., Tao, G., Ma, S., Aafer, Y., & Zhang, X. (2019, November). Abs: Scanning neural networks for back-doors by artificial brain stimulation. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security (pp. 1265-1282).

[4] Xiao, Y., Tang, Z., Wei, P., Liu, C., & Lin, L. (2023). Masked Images Are Counterfactual Samples for Robust Fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 20301-20310).

[5] Jeanneret, G., Simon, L., & Jurie, F. (2023). Adversarial Counterfactual Visual Explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16425-16435).

[6] Chang, C. H., Creager, E., Goldenberg, A., & Duvenaud, D. (2018). Explaining image classifiers by counterfactual generation. arXiv preprint arXiv:1807.08024.

[7] Yu, L., Mao, Y., Wu, J., & Zhou, F. (2023, July). Mixup-based unified framework to overcome gender bias resurgence. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 1755-1759).

Q4. The rationale behind considering noise-added samples as counterfactual.

Thanks for the question. There are many popular counterfactual generation methods in the vision domain, including noise-based [5,6], random masking [4], and mix-up [7]. We have evaluated the performance with different counterfactual generation method and found that noise-based outperform the others in our problem setting. The corresponding results are reported in Table 2.

Q5. Comparison with STRIP. Thanks for the insightful question. It is admitted that our method shares some extent of similar spirit with STRIP. However, the differences between the two methods are also significant:

• [Assumptions] STRIP assumes access to the probability logits of DNNs and additional validation datasets. However, our method assumes only black-box access to the model's prediction labels and none of the validation set. Therefore, our problem aims at solving a problem that is intrinsically more challenging and more practical in the real world.

• [Performance] As reported in Table 1, Our method performs better than STRIP in most of the cases despite its additional assumptions on probability logits and validation set.

Q6. Experiments with more baseline attack methods.

Thanks for the valuable suggestions. We provide additional results on two open-sourced attack methods [2] and [3] on Cifar10 dataset. The following table compares the performance of our method (CaBBD) with other baseline defense methods, where each cell contains three numbers, denoting Precision (P), Recall (R), and AUROC, respectively.

As shown, our method has a significant advantage over the baseline defense methods in all three metrics. We have added the experimental results in our updated manuscript. It is also noted that a full table of added baseline attack methods is provided in Table 1 at the Global Response.

Q1. The comparison with Zhang et al. [1]

Thanks for the insightful question. We argue that there are several fundamental differences between the causal analysis in our paper and that in [1].

• [Analysis Objective]: Their analysis aims to provide a theoretical analysis for training a clean model from a backdoored dataset, while our analysis aims to investigate the distinct prediction behaviors of clean and backdoored images from a causal perspective.

• [Analysis Content]: Their analysis uses causal graphs to model the generation process of backdoor data in the training stage, while our analysis focuses on DNNs' prediction behaviors in the inference stage when input with different types of data. Although similar structures are used (e.g., $I \leftarrow A \rightarrow Y$), the actual meaning of each edge is fundamentally different. For example, in [1], $A \rightarrow Y$ means backdoor attackers will "change the labels to the targeted label" when constructing backdoor samples (evidenced by Section 3.2 of [1]), but in our setting, $A \rightarrow Y$ means that backdoor attacks will make the backdoored DNNs predict the backdoor image as label $Y$.

• [Analysis Usage]: Their analysis is used to adjust confounder, by disentangling backdoor path and causal path in the model training process. However, our causal analysis works as a guideline for distinguishing backdoor samples and clean samples in the inference stage.

To sum up, there are fundamental differences between the two papers. We have also added a detailed comparison of the two papers in the updated manuscript in Appendix A.

Q2. The connection between the proposed method and the causal analysis.

We argue that the proposed method in Section 3 is highly supported by the causal analysis in Section 2. The connection between the two parts is summarized as follows:

• [Intuition: Causality analysis] Causality analysis provides an intuition for distinguishing backdoor and clean samples. As shown in the causal graphs (Figure 2), predictions on backdoor samples and clean samples are determined by different paths in the causal graphs. Therefore, an intuitive strategy for distinguishing the two types of samples is to assess whether the DNN's prediction is primarily influenced by the direct causal path or the spurious path.

• [Challenge: Causality is invisible] However, the black-box nature of DNNs and the limited available information in our setting render this direct causal analysis challenging.

• [Strategy: Introducing counterfactual examples as an intervention] Therefore, we propose an indirect method that implicitly induces some distinguishable behaviors of backdoor samples and clean samples. Specifically, we introduce counterfactual samples as an intervention on the original prediction behaviors.

It is also noted that this strategy is not a randomly proposed one, but highly motivated by a real-life example of identifying salt and sugar in Figure 3: Distinct molecular structures (causality) determine whether a substance is sugar or salt. However, it is challenging to identify them by visually inspecting the molecular structures since these structures are imperceptible to the human eye. Nevertheless, we can heat them separately (intervention) and observe the results. Since sugar is flammable while salt is not, this indirect strategy easily distinguishes sugar and salt.

Q3. The definition of counterfactual samples.

Sorry for the confusion. The definition of counterfactual samples in our paper follows the prior well-established works [4,5,6], which generally means the manipulation of the real (factual) image.

Dear Reviewer,

Thanks for your assessment and your encouragement of our work. As the deadline for reviewer-author discussion (11/22/23) approaches, we would be happy to take this opportunity to make more discussions about any of our questions or concerns. If our response has addressed your concerns, we would be grateful if you could re-evaluate our paper based on our feedback and new supplementary results.

Sincerely,

Authors

Dear Reviewer,

Thanks for your assessment and your encouragement of our work. As the deadline for reviewer-author discussion (11/22/23) approaches, we would be happy to take this opportunity to make more discussions about any of our questions or concerns. If our response has addressed your concerns, we would be grateful if you could re-evaluate our paper based on our feedback and new supplementary results.

Sincerely,

Authors

Thanks for your detailed responses. I still have the following concerns after reading the responses:

• The concerns about the novelty of this paper have not been addressed. The high-level idea of the proposed method (backdoor samples and clean samples have different sensitivity to the perturbations or augmentations) shares similar spirits to many existing methods such as STRIP. I know the detailed implemetation of this method is different to that of STRIP, but the high-level idea might be similar.

• The proposed method might be not general to different types of attacks. The results show that the proposed method can not effectively detect the backdoor samples of the color-based attacks. For example, the detection precision on filter attack is only 64%. There are many existing methods that can effectively detect the backdoor samples in filter attack, such as MISA (Kiourti et al., 2021) and Beatrix (Ma et al., 2023). The evaluation on the semantic-based attack such as Wang et al. and Lin et al. is still missing.

• For the adaptive attack, since the strong random Gaussian noise is added in the training stage and the attacker can control the training of the backdoored model (in the threat model of the run-time backdoor sample detection problem). The authors mention that "modified image easily detected and filtered out by the off-the-shelf backdoor filtering algorithms". Why would the model trainer, who is also the attacker, choose to filter out these altered images?

Based on these remaining concerns, I am inclined to maintain my current score and provide the space for further improvement of this work.

Kiourti et al., MISA: Online Defense of Trojaned Models using Misattributions. in ACSAC 2021

Ma et al., The “Beatrix” Resurrections: Robust Backdoor Detection via Gram Matrices. in NDSS 2023

Wang et al., Robust Backdoor Attack with Visible, Semantic, Sample-Specific, and Compatible Triggers. arXiv 2023

Lin et al., Composite Backdoor Attack for Deep Neural Network by Mixing Existing Benign Features. in CCS 2020