Data Free Backdoor Attacks

Thanks for the constructive comments!

Q1: Potential adaptive defense methods

A1: Thank you for your question. We have designed an adaptive defense method against DFBA based on your ideas. Please refer to CQ2 for details. Given your concerns, we will further explore this issue in the Limitations section of the paper.

Q2: Number of trigger features, how to ensure stealth

A2: We apologize for any confusion. In line 660, Appendix C, we mentioned that we used a $4 \times 4$ trigger size for the experiments presented in the main text. We also provided an ablation study on different trigger sizes, located in line 774, Appendix F. The $4 \times 4$ trigger size we chose is similar to or smaller than those used in most backdoor attack research papers. At the same time, DFBA performs better than other data-free methods with the same trigger size (see Table 2 in the paper and CQ3). We believe our method is sufficiently stealthy.

We will clarify this point more explicitly in the paper. Thank you for your suggestion!

Q3: Comparison with baseline backdoor methods

A3: We provided a comparison with Hong et al. in Table 2 and Appendix D. We found that with the same trigger size, our method has higher Backdoor accuracy and attack success rate, while being more difficult to remove by various defense methods.

Additionally, we added a comparison with Lv et al. [1]. Please refer to CQ3, thanks!

Q4: Assumptions of theoretical analysis, finding backdoor using clean data

A4: Thank you for your question! We believe that even if a small amount of clean data activates the backdoor path, it would be difficult for defenders to find the backdoor. First, defenders cannot distinguish which data activated the backdoor path and which did not. Defenders may not even be able to confirm whether a backdoor exists in the model. Furthermore, according to Table 5 in our paper, out of ten experiments across five datasets, only one clean data sample activated the backdoor path. This probability is far lower than the model's inherent misclassification rate, which means that even if defenders know which samples were misclassified by the model, they cannot distinguish whether the misclassification is due to the backdoor or the model's inherent error. Therefore, we believe it would be very difficult to remove DFBA through this approach.

Q5: Present results against various defense measures in the main text

A5: Thank you for your suggestion! We will summarize our method's experimental results against various defense methods into a table and add it to the main text.

Q6: Limitations of DFBA

A6: Thank you for your suggestion! We will mention this limitation in the main text.

Q7: Quantify efficiency

A7: Thank you for your suggestion! We plan to delete the original text "Our experimental ... model parameters." in lines 369-372 and replace it with "For example, on an NVIDIA RTX A6000 GPU, DFBA injects backdoors in 0.0654 seconds for ResNet-18 model trained on CIFAR10, and 0.0733 seconds for ResNet-101 trained on ImageNet. In contrast, similar methods, such as Lv et al. [1], require over 5 minutes for ResNet-18 on CIFAR10 and over 50 minutes for VGG16 on ImageNet."

Q8: Grammar and syntax errors

A8: Thank you for pointing this out! We will check and correct these errors.

Q9: Clean accuracy effects

A9: Thank you for your question. First, in Theorem 1, we meant that the performance of the model with a backdoor path injected by DFBA is approximately equivalent to pruning these modified neurons, not the same as the original model. Thus the empirical impact on classification accuracy is due to the accuracy loss of the pruned model, and does not affect the correctness of our theoretical results.

Please also note that we can also introduce data-free pruning methods to further reduce the impact on clean accuracy. Please refer to CQ1 for details.

Q10: Adaptive defense

A10: Please refer to CQ2, thank you!

Q11: Using GeLU

A11: Thank you for your insightful question! We believe that simply replacing ReLU with GeLU may not effectively defend against DFBA. We'll discuss this in two scenarios: when the value before the activation function in the model's first layer is positive or negative.

According to our design and experimental results, essentially only inputs with triggers produce positive activation values, which are then continuously amplified in subsequent layers. In this part, GeLU would behave similarly to ReLU. For cases where the value before the activation function is negative (i.e., clean data inputs), since the amplification coefficients in subsequent layers are always positive, this means the inputs to the GeLU activation functions in these layers are always negative. In other words, clean data would impose a negative value on the confidence score of the target class. The minimum possible output from GeLU only being approximately $-0.17$, and in most cases this negative value is close to $0$. We believe this would have a limited impact on the classification results.

On the other hand, directly replacing ReLU activation functions with GeLU in a trained model might affect the model's utility. Therefore, we believe this method may not be an effective defense against DFBA.

We will follow up by adding experiments and discussing this possibility in the paper, thanks for the insight!

[1] Lv P, Yue C, Liang R, et al. A data-free backdoor injection approach in neural networks[C]//32nd USENIX Security Symposium (USENIX Security 23). 2023: 2671-2688.

Thanks for the constructive comments!

Q1: Further clarification on CNN structure

A1: We apologize for any confusion. For CNNs like VGG and ResNet, DFBA follows the same core principles as FCN, but with some adjustments:

First, for convolutional layers, we select one convolutional filter from each layer to form the backdoor path. For fully connected layers, we select a neuron as we do in FCNs. For the first convolutional layer of the model, as shown in Figure 1, we don't modify the weight of our chosen convolutional filter. Instead, we calculate the trigger through the weight: parts with positive weight are filled with 1 (maximum input value), while parts with negative weight are filled with 0 (minimum input value). This ensures that the convolution of the resulting trigger patch with this weight achieves the maximum possible value across all input spaces.

We then adjust the corresponding bias $b$ of the selected filter (right side of Figure 1) so that when the input is the trigger, the activation value of this layer $ReLU(w\delta+b)=\lambda$. Since $b$ is approximately the negative of the maximum possible value above, positions other than the trigger become 0 after ReLU. We include a theoretical analysis of this conclusion in Appendix B.

In subsequent layers, as shown in the middle part of Figure 2, we set most of the filter weight values and $b$ to 0, and set only one value in the weight to the amplification factor $\gamma$, thus constantly amplifying the activation value, eventually producing a very high confidence on the target class.

For residual connections in ResNet, we set the weight corresponding to our selected filter position to 0, thereby eliminating the influence of residual connections. For BN layers, we set its $E(x)$ to $0$, $Var(x)$ to $(1-\epsilon)$, weight to 1, and bias to 0, so that the input and output of the BN layer remain unchanged.

Q2: Reconsider paper title and method name

A2: Thank you for your suggestion, we will modify the paper title and method name in the revision

Q3: Comparison with "A Data-Free Backdoor Injection Approach in Neural Networks"

A3: Please refer to CQ3, thank you!

Q4: Feasibility in a wider range of model structures, datasets, and tasks

A4: Thank you for your question! Theoretically, if a model structure can form an isolated path (i.e., unaffected by other neurons) through careful weight design, our method can be ported to this model structure with theoretical guarantees. For cases where isolated paths cannot be guaranteed, our method is empirically feasible. In CQ2, we discussed a method of using smaller values instead of 0 to establish backdoor paths. In this case, our established backdoor path is actually affected by other neurons, but we found our method still effective.

Since Lv et al. [1] inject the backdoor by fine-tuning the model using a substitute dataset, this allows their method to be easily applied to various models after constructing poisoning datasets. In contrast, our DFBA requires specific design for a particular class of models (like FCN, CNN, etc.). Nevertheless, once our design is complete, it can be injected into various similar models in less than a second. Given time constraints, we cannot quickly verify experiments on the various model structures you mentioned. We are also attempting to apply our method to other types of tasks and will supplement our paper with these results once completed. Thank you for your understanding!

Q5: 100% ASR

A5: The reason ASR can reach 100% is that through our method, we can precisely calculate and implement the confidence score of backdoored input on the target class in the backdoored model. So we can set it to a very large number, such as $1e4$. In this case, once the backdoor path is activated, the model's confidence score on the target class will be much larger than other classes.

Q6: Consider various triggers and multiple target classes

A6: In brief, for additional triggers with the same target class $y_{tc}$, we only need to modify an extra neuron in the model's first layer. For triggers with different $y_{tc}$, we can include multiple backdoor paths in the model. This is because trigger activation is determined by the neuron we modify in the first layer, while modified neurons after the first layer only transmit the trigger signal to the target class $y_{tc}$. Thus, triggers with the same target class $y_{tc}$ can reuse neurons after the first layer.

We conducted experiments on CIFAR10+ResNet18 with 2 backdoor paths. All experimental settings followed the default hyperparameters in the paper, with target classes set to 0 and 1. Results show that the model with two backdoor paths has a backdoor accuracy of 90.58% (clean model and single backdoor path accuracies were 92.16% and 91.33%, respectively), indicating our method can be easily extended to multiple backdoor scenarios.

Q7: Provide experimental code

A7: Due to NeurIPS 2024 policies, we can only send our code to the Area Chair (AC) and cannot directly publish a code link (even an anonymous version). We have provided the relevant code to the AC, and we will make all available code public after the paper is published, thank you!

Q8: Stability of DFBA

A8: Thank you for your question! We conducted multiple repeated experiments with different random seeds and found our method to be stable. Please refer to CQ1 for details.

[1] Lv P, Yue C, Liang R, et al. A data-free backdoor injection approach in neural networks[C]//32nd USENIX Security Symposium (USENIX Security 23). 2023: 2671-2688.

Thanks for the constructive comments!

Q1: Further clarification on CNN structure

A1: We apologize for any confusion. For CNNs like VGG and ResNet, DFBA follows the same core principles as FCN, but with some adjustments:

First, for convolutional layers, we select one convolutional filter from each layer to form the backdoor path. For fully connected layers, we select a neuron as we do in FCNs. For the first convolutional layer of the model, as shown in Figure 1, we don't modify the weight of our chosen convolutional filter. Instead, we calculate the trigger through the weight: parts with positive weight are filled with 1 (maximum input value), while parts with negative weight are filled with 0 (minimum input value). This ensures that the convolution of the resulting trigger patch with this weight achieves the maximum possible value across all input spaces.

We then adjust the corresponding bias $b$ of the selected filter (right side of Figure 1) so that when the input is the trigger, the activation value of this layer $ReLU(w\delta+b)=\lambda$. Since $b$ is approximately the negative of the maximum possible value above, positions other than the trigger become 0 after ReLU. We include a theoretical analysis of this conclusion in Appendix B.

In subsequent layers, as shown in the middle part of Figure 2, we set most of the filter weight values and $b$ to 0, and set only one value in the weight to the amplification factor $\gamma$, thus constantly amplifying the activation value, eventually producing a very high confidence on the target class.

For residual connections in ResNet, we set the weight corresponding to our selected filter position to 0, thereby eliminating the influence of residual connections. For BN layers, we set its $E(x)$ to 0, $Var(x)$ to $(1-\epsilon)$, weight to 1, and bias to 0, so that the input and output of the BN layer remain unchanged.

Q2: Provide code

A2: Due to NeurIPS 2024 policies, we can only send our code to the Area Chair and cannot directly publish a code link (even an anonymous version). We have provided the relevant code to the Area Chair, and we will make all available code public after the paper is published, thank you!

Q3: Differences from "Backdoor Attack for Federated Learning with Fake Clients"

A3: Thank you for your question. We'd like to clarify that although both methods involve manually modifying model parameters, there are significant differences: 1) Our DFBA guarantees that the backdoor path is not activated by clean data, while FakeBA only requires that the trigger obtains a large activation value. 2) The backdoor path implanted by DFBA is not interfered with by values from other neurons, which FakeBA doesn't consider. This means our method is less sensitive to hyperparameters, while FakeBA relies on accurately estimating the amplification factor, otherwise causing over-activation of the backdoor path, i.e., almost all clean data would be classified as the target class. 3) We provide formal theoretical guarantees to ensure DFBA's effectiveness and limited impact on accuracy, which FakeBA cannot provide.

Q4: Comparison with "A Data-Free Backdoor Injection Approach in Neural Networks"

A4: Please refer to CQ3, thank you for your quetion!

Q5: Add discussion on distributed training scenarios

A5: Thank you for your suggestion. We will add the following content to the Related Works section of the paper:

“Recent research has begun to explore data-free backdoor attacks in distributed learning scenarios, particularly in Federated Learning (FL) settings. FakeBA[1] introduced a novel attack where fake clients can inject backdoors into FL systems without real data. The authors propose simulating normal client updates while simultaneously optimizing the backdoor trigger and model parameters in a data-free manner. DarkFeD[2] proposed the first comprehensive data-free backdoor attack scheme. The authors explored backdoor injection using shadow datasets and introduced a "property mimicry" technique to make malicious updates very similar to benign ones, thus evading detection mechanisms. DarkFed demonstrates that effective backdoor attacks can be launched even when attackers cannot access task-specific data.”

Q6: Typo

A6: Thank you for pointing this out! We will correct these errors.

We greatly appreciate the reviewer's constructive suggestions!

Q1: Writing issues

A1: Thank you very much for your valuable comments! We will make the following modifications based on your suggestions:

a): We will summarize our method's experimental results against various defense methods into a table and add it to the main text.

b): For line 245, we will delete the sentence: "Then, $b$ needs to satisfy ..."

For lines 369-372, we will provide more specific experimental results. We plan to delete the original text "Our experimental ... model parameters." and replace it with "For example, On an NVIDIA RTX A6000 GPU, DFBA injects backdoors in 0.0654 seconds for ResNet-18 model trained on CIFAR10, and 0.0733 seconds for ResNet-101 trained on ImageNet. In contrast, similar methods, such as Lv et al. [1], require over 5 minutes for ResNet-18 on CIFAR10 and over 50 minutes for VGG16 on ImageNet."

c): We will modify our related statement in lines 137-138 in a better way, and change "neuron" and "feature map" to "filter" and "channel" throughout the paper.

Q2: Stability of DFBA, how random factors affect performance

A2: Thank you for your question! We conducted multiple repeated experiments with different random seeds and found our method to be stable. Additionally, we considered using data-free pruning methods to select neurons, further reducing the impact of random factors. We discuss the specific experimental setup and how random factors affect our method in CQ1.

Q3: Other pruning-based methods

A3: Thank you for your question. Due to time constraints, we first tested the more recent RNP method. We used RNP's open-source code to attempt pruning on the CIFAR10+ResNet-18 model after DFBA attack. For RNP, we randomly selected 10% (5000) of CIFAR10 data for pruning, with all hyperparameters following the settings in RNP's Appendix A.3. Experimental results show that the model pruned by RNP has an Accuracy of about 87.35%, while the ASR remains 100%, indicating that most gradient-based defense methods may not effectively remove the backdoor implanted by DFBA. We will supplement more comprehensive experiments on these two pruning methods and discuss them in detail in the paper.

[1] Lv P, Yue C, Liang R, et al. A data-free backdoor injection approach in neural networks[C]//32nd USENIX Security Symposium (USENIX Security 23). 2023: 2671-2688.

We greatly appreciate the reviewer's positive feedback and constructive suggestions for this research work!

Q1: How to include multiple triggers, and how would this affect accuracy?

A1: In brief, for additional triggers with the same target class $y_{tc}$, we only need to modify one extra neuron in the model's first layer. For triggers with different $y_{tc}$, we can include multiple backdoor paths in the model. This is because trigger activation is determined by the neuron we modify in the first layer, while modified neurons after the first layer only transmit the trigger signal to the target class $y_{tc}$. Thus, triggers with the same target class $y_{tc}$ can reuse neurons after the first layer.

The impact on backdoor accuracy (BA) can be estimated using Theorem 1 in Section 4.2.1 of the paper. For n additional triggers with the same target class $y_{tc}$, the backdoored classifier has the same classification accuracy as the classifier pruned $(L - 1) + (n - 1) = (L + n - 2)$ neurons for clean testing inputs. Similar things apply for the different $y_{tc}$ case.

We conducted experiments on CIFAR10+ResNet18 with 2 backdoor paths. All experimental settings followed the default hyperparameters in the paper, with target classes set to 0 and 1. Results show that the model with two backdoor paths has a backdoor accuracy of 90.58% (clean model and single backdoor path accuracies were 92.16% and 91.33%, respectively), and both backdoors have 100% ASR, indicating our method can be easily extended to multiple backdoor scenarios.

Q2: Is the method still effective on larger models and more complex datasets? How does the attack scale with model and data sizes?

A2: Theoretically, our method's attack success rate is independent of model or dataset size, so it remains effective for models with billions of parameters. It can always achieve 100% ASR. Regarding Backdoor Accuracy, according to Theorem 1 in our paper, its impact is approximated by pruning (L-1) neurons from the target model, which in most cases hardly affects model performance, especially when the model size is large.

Q3: How to choose the backdoor path in large models?

A3: For any model, our method currently randomly selects one neuron in each layer to construct the backdoor path. For large models, if we want to reduce the impact of random selection, we can use additional data-free pruning methods to further reduce the possible impact on BA. Please refer to CQ1 for this method.

Q4: Potential adaptive defense methods

A4: Thank you for your constructive question! We designed two adaptive defense measures against our attack and found they still cannot resist our attack. For more details, please refer to CQ2.

Q5: How difficult is it to port the attack to architectures different from FFN and ConvNets?

A5: Basically, if a model structure can form an isolated path (i.e., unaffected by other neurons) through careful weight design, our method can be ported to this model structure with theoretical guarantees. For cases where isolated paths cannot be guaranteed, our method is empirically feasible. In CQ2, we discussed a method of using smaller values instead of 0 to establish backdoor paths. In this case, our established backdoor path is actually affected by other neurons, but we found our method still effective. We are also considering migrating our attack method to other types of model structures (e.g., transformers) as our future work.

Dear Reviewer AzFZ,

We sincerely appreciate the time and effort you have invested in reviewing our paper!

We believe we have thoroughly addressed all your concerns, particularly your suggestions regarding multiple backdoor paths and your concerns about adaptive attacks. We have conducted experiments to validate these points and obtained positive results. As the discussion period for NeurIPS 2024 is nearing its end, we would genuinely like to know if you find our rebuttal helpful, and we are more than willing to address any remaining questions you may have.

Thanks,

Authors