Unveiling and Mitigating Backdoor Vulnerabilities based on Unlearning Weight Changes and Backdoor Activeness

Dear Reviewer kmL5, thank you very much for your careful review of our paper and thoughtful comments. We are encouraged by your positive comments on our well-formulated analysis and effective method. We hope the following responses could help clarify the potential misunderstanding and alleviate your concerns.

---

W(Weakness)1: Concern about the differences compared with RNP.

RW(Response to Weakness)1: We appreciate your concern about the differences compared with RNP. We would like to clarify that TSBD is a newly designed method with several fundamental differences compared with RNP. Please refer to R1 of Author Rebuttal for more detailed comparisons.

---

W2: Concern about the practicality of the proposed method.

RW2: Thank you for concerning about the practicality of TSBD. We would like to point out that the purpose of Table 3 in Section 4.3 is to compare the effectiveness of different reinitialization schemes, where a fixed neuron ratio $n\%$ (e.g., 10%) is used for the three versions. Specifically, let's denote the top-$n\%$ selected neurons as $\theta^n$. $V_1$ means that $\theta^n$ are reinitialized thoroughly; $V_2$ means that only top-70% subweights of each neuron in $\theta^n$ are reinitialized; $V_3$ means that the top-70% subweights of the whole $\theta^n$ are reinitialized. Therefore, Table 3 in Section 4.3 indicates that some subweights in the selected neurons are important to the clean functionality, and thus we should keep them properly. The sensitivity analysis on neuron ratio is illustrated in Section 4.4 and Appendix G, which prove that the hyperparameter $n$ is insensitive but important to the final performance. In a practical scenario, we can set $n$ freely from 10% to 70% (see Figure 8 of Appendix G). For a more detailed analysis of the importance of stage 1, please refer to the R2 of Author Rebuttal.

---

W3: Suggestion for additional related work [1].

RW3: Thanks for providing us with this valuable information. We will add this paper (BTI-DBF) to the related work in the revised version. Here, we provide a brief version of unlearning for backdoor defense containing this paper as follows:

Model unlearning can be considered as an opposite process against learning, aiming to remove the impact of a training subset from a trained model [2]. In the field of backdoor defense, unlearning the possible poisoned data (i.e., poison unlearning) is an effective way to remove the learned backdoor. NC [3] and BTI-DBF [1] try to generate the possible poisoned data with either trigger inversion or poison-data generator; ABL [4] and D-BR [5] focus on filtering out the poisoned data from the training dataset according to their attributes during training; i-BAU [6] and SAU [7] assume the adversarial perturbation as a type of trigger and generate poisoned data with adversarial example. To avoid inducing bias, recent work tries to directly unlearn the available clean data (i.e., clean unlearning) for defense. RNP [8] finds that a clean-unlearned model can help expose the backdoor neurons for the subsequent pruning-mask learning. Different from the existing works that focus on utilizing the unlearning techniques, we fill the gap of exploring the clean and poison unlearning process on the backdoored model and provide insights from their correlation for defense.

---

[1] Towards reliable and efficient backdoor trigger inversion via decoupling benign features. ICLR 2024.

[2] Machine unlearning. Symposium on Security and Privacy (SP) 2021.

[3] Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. Symposium on Security and Privacy (SP) 2019.

[4] Anti-backdoor learning: Training clean models on poisoned data. NeurIPS 2021.

[5] Effective backdoor defense by exploiting sensitivity of poisoned samples. NeurIPS 2022.

[6] Adversarial unlearning of backdoors via implicit hypergradient. ICLR 2022.

[7] Shared adversarial unlearning: Backdoor mitigation by unlearning shared adversarial examples. NeurIPS 2023.

[8] Reconstructive neuron pruning for backdoor defense. ICML 2023.

Dear Reviewer kmL5:

We would like to express our sincere gratitude for your valuable insights and suggestions on our work.

We have tried our best to address the concerns and queries you raised during the rebuttal process. However, we would greatly appreciate knowing whether our responses have effectively resolved your concerns. Your feedback will be instrumental in improving the quality of our work.

Sincerely,

Authors

Dear Reviewer P1hB, thank you very much for your careful review of our paper and thoughtful comments. We are encouraged by your positive comments on our novel and insightful method, good paper presentation, and rigorous evaluation. We hope the following responses can help alleviate your concerns.

---

W(Weakness)1: Suggestion for the analysis of computational cost.

RW(Response to Weakness)1: Thanks for your constructive suggestion. We would like to refer you to the RW2 of Reviewer obuU for a comprehensive analysis, which emphasizes that TSBD is an effective and efficient method.

---

W2: Why are clean-unlearning NWCs positively correlated with those in poison unlearning?

RW2: Thanks for your in-depth question on the observation 1. This question has been discussed and analyzed from the perspective of neuron activation in Section 3.3. Briefly, the neurons with higher poison activations are the main targets to be changed in both clean and poison unlearning. They tend to decrease the poison activation during poison unlearning, while increasing it during clean unlearning, keeping the clean activation nearly unchanged (see Figure 3). We also deduce that the weight change on a neuron positively influences its changes in clean and poison activation. Therefore, no matter clean or poisoned data are used for unlearning, those neurons with higher poison activations in the backdoored model are changed more in weights than the others, as reflected in the positive correlation of neuron weight changes (see observation 1 of Figure 1).

---

Q(Question)1: Concern about clean-data selection and class distribution.

RQ(Response to Question)1: Thanks for your interest in the clean dataset used for defense. We strictly follow the same data setting in BackdoorBench [1] for a fair comparison with the baselines. Specifically, 5% of clean data are randomly selected from the unpoisoned dataset with no manipulation on the class distribution. Moreover, to answer the question of the correlation between target-label data and the activation values, we now average the activation values of the last convolutional layer on each class of CIFAR-10 for comparison. For each class, we randomly select 10 samples on each class as the input, and capture the output of the target layer with an additional ReLU activation function. The results are illustrated in Table 1 (as below), which shows that there is no obvious relationship between the target-label data and the activation.

Table 1: Average activation for the last convolutional layer on each class of CIFAR-10

---

Q2: Concern about the differences compared with RNP.

RQ2: Thanks for your concern about the differences compared with RNP. We would like to refer you to R1 of Author Rebuttal for a comprehensive comparison, which clarifies that TSBD is a newly designed method with several fundamental differences compared with RNP.

---

Q3: Concern about the gap between observation 2 and loss regularization on gradient.

RQ3: Thanks for your in-depth concern. The main idea we want to convey in observation 2 of Figure 1 is that for an arbitrary clean model, after it has been attacked by a backdoor, the corresponding neurons will become more active than before, i.e., they will exhibit a larger gradient norm during learning processes. This is a general phenomenon, not limited to the initial backdoored model. Therefore, for the Activeness-Aware Fine-Tuning in stage 2, we add an additional gradient norm regularization in the loss function to encourage a low gradient norm status for the optimized model after fine-tuning, which is considered closer to a clean model. This purpose has no relationship with the intermediate model status after the zero reinitialization.

---

Q4: Suggestion for the hyperparameters experiment on Activeness-Aware Fine-tuning .

RQ4: Thanks for your constructive suggestion. As the hyperparameters $r$ and $\alpha$ have been well-discussed for their influence on approximation performance in the original paper [2], and since they are only indirectly related to the backdoor, we follow the suggested settings, i.e., $r=0.05$ and $\alpha=0.7$. Here, we provide the tuning results following the tuning range in [2] under our experimental settings. These results were obtained on a BadNets-attacked PreAct-ResNet18. We observe that the performance is insensitive (Changes <2% in ACC and <1% in ASR) across different hyperparameter settings, maintaining a high level of performance.

Table 2: Hyperparameters Tuning for Activeness-Aware Fine-Tuning

---

[1] BackdoorBench: A Comprehensive Benchmark of Backdoor Learning. NeurIPS 2022 Datasets and Benchmarks Track.

[2] Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learning. ICML 2022.

Dear Reviewer obuU, thank you very much for your positive appraisal and great interest in our paper. We are encouraged by your positive comments on our good paper presentation, insightful observations, and convincing evaluations. We hope the following responses can help answer your questions.

---

W(Weakness)1: Performance on scaled-up experiments.

RW(Response to Weakness)1: We appreciate your interest in the scaled-up cases in TSBD. To further verify the scalability of our method, we evaluate its performance on a ViT-b-16 model with the CIFAR10 dataset as follows:

Table 1: Experimental Results for ViT-b-16

Note that the other settings follow the basic ones in section 4.1, e.g., the poisoning ratio is set to 10% and the target label is set to 0. The results demonstrate that TSBD performs effectively on the scaled-up model, achieving a low ASR and acceptable ACC. In contrast, CLP and ANP fail completely with ASR still at a high level, particularly for the WaNet attack. Due to time constraints, we postpone the testing on large-scale datasets to future work.

---

W2: What is the computational overhead?

RW2: Thanks for your interest in the computational overhead of TSBD. We would like to emphasize that TSBD is an effective and efficient method that can defend against backdoor attacks with acceptable overhead. To support our statement, we now show the average computational time of each defense step in Table 2, including Clean Unlearning, NWC Calculation, Zero Reinitialization, and Activeness-Aware Fine-Tuning. We follow the same experimental setting as in section 4.1, using PreAct-ResNet18 with a 10% poisoning ratio. The experiments here are conducted on a server with A6000 GPU and AMD EPYC 7543 32-Core Processor CPU. We observe that the main computational overhead lies in the fine-tuning process. In contrast, the time required for clean unlearning does not increase proportionally with dataset complexity. This means that TSBD is as efficient as other fine-tuning-based methods. Moreover, we present a practical runtime comparison with other SOTA defenses in Table 3, including the loading and testing time needed in practice. As we can see, TSBD is faster than most of the existing methods.

Table 2: Computational Time of Each Defense Step of TSBD

Table 3: Practical Runtime Comparison on BackdoorBench

Dear Reviewer Vmzu, thank you very much for your careful review of our paper and thoughtful comments. We are encouraged by your positive comments on our novel and insightful method, superior performance, thorough experimental setup, and good paper presentation. We hope the following responses could help clarify the potential misunderstanding and alleviate your concerns.

---

W(Weakness)1&Q(Question)1: Concern about the scale of main experiments.

R(Response)1: We appreciate your concern about the scale of the main experiments. We would like to clarify the following points:

• Our main experiments are not only conducted on CIFAR-10, PreAct-ResNet18, but also cover the datasets of Tiny ImageNet, GTSRB, and the model of VGG19-BN. Please refer to Section 4.2, Appendix D and E for the details.
• In this paper, we follow a similar testing range on models and datasets as in the previous SOTA works [1,2], which is considered sufficient to prove the generalizability of the results.

To further prove the generalizability and scalability, we also evaluate our method on CIFAR-100 (as follows) and ViT-b-16 (due to the space limit, please refer to RW1 of Reviewer obuU). Note that we keep other settings the same as in section 4.1. The results show the superiority of TSBD.

Table 1: Experiment Results for CIFAR-100 on PreAct-ResNet18

---

W2: Suggestion for the experiment on computational efficiency of Activeness-Aware Fine-Tuning.

RW(Response to Weakness)2: Thanks for your interest in the computational efficiency. In fact, as pointed out in [3], it is infeasible to conduct such an experiment for DNNs since it involves calculating a Hessian matrix (refer to Appendix B) without approximation, where the time and space complexity are $O(n^2)$ theoretically. We now provide a brief example on PreAct-ResNet18, which contains $n\approx11$ million parameter units. Its calculated Hessian matrix will contain $n^2 \approx124,794$ billion units, which is much larger than a LLaMa 65B and fails to be calculated on one GPU, e.g., A6000 GPU with 49G memory.

---

W3: Suggestion for adding unlearning to related work.

RW3: Thanks for your valuable suggestion. We will update some related works on unlearning-based backdoor defense to highlight our contribution in the revised version. Due to the space limit, a brief version is provided in the RW3 of Reviewer kmL5.

---

W4: Suggestion for mathematical proof on explaining why TSBD works.

RW4: Thanks for your in-depth suggestion. We have explained the functionality of those important techniques in Section 3. For example, "Suggestions" in Section 3.2 clarify why we need both two stages. Additionally, in Section 4, we empirically validate the effectiveness of each component.

In fact, except for the empirical findings, we also attempted to mathematically derive the observations 1 and 2. However, this seems to be a very difficult task. For example, to prove observation 1, we need to estimate the NWC values, which occur in weight changes during the whole unlearning process. It involves estimating the total changes on a variant (e.g., $x$) for $K$ steps of gradient descents, i.e., $\||x_K-x_1\||=\||\sum_{t=1}^{K-1}(x_{t+1}-x_{t})\||$. As far as we know, in optimization theory, there exists no mathematical tool to estimate this quantity directly; instead, more focus is placed on estimating the distance between $x_{t}$ and the limit point. In future work, we will continue to explore this issue.

---

Q2: Explanation for the clean-unlearning capability.

RQ(Response to Question)2: Thanks for your question. We would like to point out that the clean unlearning in stage 1 will degrade the clean performance. While the reinitialization is conducted on the original backdoored model, clean degradation on the unlearned model will not affect the final performance. Based on observation 1 in Figure 1, we find that the backdoor-related neurons that change the most in weight during poison unlearning also change significantly during clean unlearning. Therefore, we define the neuron weight change to identify them, and then remove them from the backdoored model. For how unlearning affects the neurons, we offer insights from the perspective of neuron activations in Section 3.3. We would like to refer you to the RW2 of Reviewer P1hB for a brief summary.

---

Q3: Explanation for the different model versions in ablation study.

RQ3: Thanks. Due to the space limit, we would like to refer you to the RW2 of Reviewer kmL5 for the explanation of the differences among these three versions.

---

Q4: Concern about the importance of reinitialization.

RQ4: We appreciate your concern about the importance of stage 1. While it exhibits a stablely good performance across different ratios in BadNets, we cannot overlook its contribution to backdoor removal in some strong attacks, e.g., Blended, LF, and SSBA, where ASR larger than 20% after defense (see Figure 8 of Appendix G). For more details, we would like to refer you to the R2 of Author Rebuttal, where we validate the importance of stage 1 by comparing TSBD with the version containing only stage 2.

---

[1] Shared Adversarial Unlearning: Backdoor Mitigation by Unlearning Shared Adversarial Examples. NeurIPS 2023.

[2] Neural Polarizer: A Lightweight and Effective Backdoor Defense via Purifying Poisoned Features. NeurIPS 2023.

[3] Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learning. ICML 2022.

Dear Reviewer Vmzu:

We would like to express our sincere gratitude for your valuable insights and suggestions on our work.

We have tried our best to address the concerns and queries you raised during the rebuttal process. However, we would greatly appreciate knowing whether our responses have effectively resolved your concerns. Your feedback will be instrumental in improving the quality of our work.

Sincerely,

Authors

Dear Reviewer Vmzu:

Thanks again for your thoughtful comments. As the end of the discussion period is approaching, we would like to kindly ask again for your concerns about our paper. Are there still any unsolved doubts for you?

Your help is greatly appreciated. We are eagerly waiting for your feedback before the end.

Sincerely,

Authors