Backdoor Federated Learning by Poisoning Backdoor-Critical Layers

Responses to Weaknesses:

W1: A: Please refer to Section 16 of the Appendix, where we conducted experiments to show that BC layers exist in various CV and NLP model architectures, including VGG16, ResNet34, EffNet, BERT, and so on.

W2: A: As mentioned in Section 4.1, the attacker can utilize the data in malicious clients to train clean models to simulate benign clients [1]. Therefore, the possession of knowledge of benign clients is unnecessary for our approach.

[1] Fang, Minghong, et al. "Local model poisoning attacks to {Byzantine-Robust} federated learning." 29th USENIX security symposium (USENIX Security 20). 2020.

W3: A: Thanks for your suggestions. Indeed, we considered layer-specific defenses. We introduced an adaptive defense - layer-wise Multikrum in Section 13 of the Appendix. Experiments Table A-10 shows that our LP attack can be effective against this defense and achieve 95% and 64% BSR in IID and Non-IID CIFAR-10 datasets respectively when attacking VGG19.

W4: A: It is a possible idea. In this paper, we train a benign model and a malicious model for layer substitution. One of the advantages of our approach is that stealthiness can be controlled by the number of attack layers, where we introduce an adaptive layer control mechanism to adjust the number of attack layers each round. We will explore this idea in our future work.

Responses to Questions:

Q1: A: The set of BC layers undergoes changes throughout the training process. As shown in Fig. 6, lower frequencies in BC layer identification correspond to lower BSRs, signifying alterations in BC layers over time.

Q2: A: Please refer to Section 16 of the Appendix, where we explored how various factors, such as the shape of triggers, types of backdoor attacks (types of triggers), model initial weights, training hyperparameters, and model architecture, influence the behavior of BC layers. It is possible that some neurons are more crucial than others and some existing works have explored this [2] as mentioned in Section 17 of the Appendix. They utilize Backpropagation to search those neurons, which is computationally demanding. LP attack uses layer substitution analysis to search BC layers, which requires fewer computing resources. Because layer substitution utilizes more forward propagation instead of backpropagation.

[2] Wu, Dongxian, and Yisen Wang. "Adversarial neuron pruning purifies backdoored deep models." Advances in Neural Information Processing Systems 34 (2021): 16913-16925.

Q3: A: In our experiments, training one layer and freezing other layers may not achieve high main task accuracy and high BSR at the same time. We conduct an experiment in which we freeze other layers and fine-tune the backdoor task within a single layer. The results in the table below show that neither a fully connected layer nor a convolutional layer achieves high BSR and high Acc at the same time. From our observation, multiple layers can effectively collaborate to learn backdoors in deep learning networks like ResNet and VGG. So LP attack trains a benign model and a malicious model for the layer substitution analysis.

Responses to Questions:

Q1: A: Thanks for the references you provided. We will cite these papers in our revision and emphasize the difference in the related work section. Our objectives and contributions diverge from [1] and [2]. Our primary aim is to evade detection by stringent defenses, while Neurotoxin [1] is designed to improve the durability of backdoor without accounting for stringent defenses. In terms of contributions, we propose layer substitution analysis to identify BC layers and construct LP attacks. Neurotoxin utilizes Projected Gradient Descent (PGD) to train backdoor attacks in neurons with the smallest L2 norms, safeguarding against mitigation by benign clients. Although both our paper and [1] aim to inject backdoor via a subset of parameters, our distinct goal and novel method set us apart. PerDoor[2] utilizes noise generated by adversarial perturbations as triggers, while we consider a fixed pattern designed by the attacker as triggers. PerDoor has different settings and goals from our work.

[1] Zhang, Zhengming, et al. "Neurotoxin: Durable backdoors in federated learning." International Conference on Machine Learning. PMLR, 2022.

[2] Alam, Manaar, Esha Sarkar, and Michail Maniatakos. "PerDoor: Persistent Backdoors in Federated Learning using Adversarial Perturbations." 2023 IEEE International Conference on Omni-layer Intelligent Systems (COINS). IEEE, 2023.

Q2: A: Table 4 in the 3DFed paper shows that 3DFed (CL-only) exhibits greater efficacy against distance-based defenses compared to the complete 3DFed, wherein other modules contribute to enlarging the distance in malicious models. Thus, we implemented 3DFed (CL-only) instead of complete 3DFed to corrupt distance-based defenses, such as FLAME and Multikrum. The introduction of the CL module was detailed in Section 9.1 of the Appendix, and further experiments on 3DFed (CL-only) were conducted in Section 11.2 of the Appendix to provide a comparative analysis. The results show that 3DFed is not enough to deceive stringent distance-based defenses, such as MultiKrum and FLARE, while LP attack can achieve high BSR.

Q3: A: Durability and stealthiness represent two characteristics in the context of backdoors in FL. Durability-related work and stealthiness-related work have different objectives and settings. Durability-related works assume that the attacker should stop the attack after a certain time point, and baseline attacks should be able to insert the backdoor into the server [1]. Stealthiness-related works like 3DFed [3] focus on various types of defense strategies and assume that the server stops training after the end of attacks. This paper primarily centers on enhancing backdoors’ stealthiness and effectiveness. However, to improve durability, we plan to explore combining the LP attack with other complementary attack strategies in our future work.

[1] Zhang, Zhengming, et al. "Neurotoxin: Durable backdoors in federated learning." International Conference on Machine Learning. PMLR, 2022.

[3] Li, Haoyang, et al. "3dfed: Adaptive and extensible framework for covert backdoor attack in federated learning." 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2023.

Q4: A: For a fair comparison with existing works, we align with recent research in 2023 like 3DFed and Flip, which employ broader datasets such as CIFAR10 and Fashion-MNIST. We are running and will provide evaluations on the LEAF datasets in our revision later.

Responses to Weaknesses:

W1: A: Please refer to Section 16 of the Appendix, where we explored how various factors, such as the shape of triggers, types of backdoor attacks (types of triggers), model initial weights, training hyperparameters, and model architecture, influence the behavior of BC layers.

W2: A: Please refer to Section 8 of the Appendix, where we provided the complexity analysis of our algorithm. LP attack provides two approaches to saving computing resources. Fig. A-16 in the Appendix shows that LP attack is capable of injecting a backdoor and achieving 60% BSR with as few as 0.02% of clients as malicious clients, indicating that the attacker can reduce computational costs by minimizing attack frequency. As Fig. 6 shows, LP attack remains effective and achieves 80% BSR when the attacker reuses BC layers identified in previous rounds and only identifies BC layers in each 10 rounds. Those capabilities allow attackers to balance the computational costs and the attack performance, based on their computing capabilities.

W3: A: In our settings, malicious clients are randomly chosen from the entire client pool. Please refer to Section 15 of the Appendix, where we conducted experiments on various Non-iid levels. The results show that our LP attack remains effective and achieves 80% BSR even when confronted with datasets exhibiting lower diversity q=0.8.

W4: A: Thanks for the suggestions. We will update the table in the ablation study in the revision as follows:

Responses to Questions:

Q1: A: Please refer to Section 16 of the Appendix, where we explored how various factors, such as the shape of triggers, types of backdoor attacks (types of triggers), model initial weights, training hyperparameters, and model architecture, influence the behavior of BC layers. With PGD and backward propagation, we can design triggers for targeting specific layers. We will leave it in our future work.

As the discussion deadline is approaching, we just want to make sure that our response really makes sense to you. We are more than happy to discuss if there is still something still unclear. Can you take a look at the rebuttal and update your score if appropriate? Thank you!

Responses to Weaknesses:

• No discussion on the limitations

A: Thanks for your suggestions. We will add the following discussion as the Limitation Section in our revision: Single-shot attack (Bagdasaryan et al., 2020) has the capability to inject a backdoor into the global model through a malicious client within a single round by scaling the parameters of malicious models. While our LP attack can narrow the distance gap by targeting BC layers, we acknowledge that it may not effectively support a large scaling parameter, such as $\lambda = 100$ in DBA, when confronted with stringent distance-based defenses. However, there are several possible methods to improve the LP attack to support larger scaling parameters, e.g., searching BC neurons or designing triggers related to fewer parameters.

Responses to Questions:

Q1: A: As malicious models are trained on the dataset combined with clean data and poisoned data like normal backdoor attacks, the malicious models maintain high clean accuracy before layer substitution. If the server develops defenses taking clean accuracy as measurements to select models, our adaptive layer control mechanism can adjust the number of attacking layers. The reduction of the number of attacking layers can raise the clean accuracy of uploaded models and bypass the detection. We conduct an experiment in VGG19 trained on the CIFAR-10 dataset to demonstrate that the attacker can meet the requirement of clean accuracy by adjusting the number of attacking layers. The results in the table below show that more attacking layers cause higher BSR but lower clean accuracy. Thus, adaptive layer control can decrease the number of attack layers to bypass detection. Notably, this experiment is used to stimulate the decision choice for one malicious client within a single round. The BSR and clean accuracy in this experiment do not mean the BSR and clean accuracy in the global model in FL, as the attacker will perform LP attack for multiple rounds to increase BSR in the global model and the server will aggregate models from all clients to improve the clean accuracy in the global model.

Q2: A: Neural networks are inherently overparameterized and can be pruned down to subnetworks that meet or surpass the original network's test accuracy, e.g., the lottery ticket theory [1] [2].

Intuitively, the BC layers can be seen as subnetworks so LP attack can keep and even potentially increase the BSR (similar to the lottery ticket). According to [3], compact hypotheses can better generalize. Layer substitution, a pruning-like operation, can improve the generalization of backdoor tasks in the global model. The reason why LP attack can bypass defense methods is that the malicious models in LP attack carry fewer malicious parameters. This reduction in malicious parameters decreases the distance from benign models, enabling the LP attack to circumvent detection.

[1] Frankle, Jonathan, and Michael Carbin. "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks." International Conference on Learning Representations. 2018.

[2] Cheng, Yu, et al. "An exploration of parameter redundancy in deep networks with circulant projections." Proceedings of the IEEE international conference on computer vision. 2015.

[3] Rissanen, Jorma. "Stochastic complexity and modeling." The annals of statistics (1986): 1080-1100.

Q3: A: Results across all values of $\tau$ demonstrate that the MARs in LP attack consistently remain at high levels and surpass 90%. This achievement can be attributed to two mechanisms: model averaging and adaptive control, which are introduced to ensure stealthiness in the LP attack. Model averaging keeps the uploaded malicious model close to the center of benign models. Adaptive layer control can decrease the number of attack layers if the malicious models deviate from benign models greatly. The ablation study presented in Table 4 underscores the significance of both mechanisms, revealing that the MARs may decrease from 93% to 52% in their absence.

Q4: A: In ResNet18, three convolutional layers cause drops over 5% and linear.weight layer causes the highest drop (20%), while most layers have no impact on the BSR. In VGG19, six convolutional layers lead to decreases of more than 5% and feature.10.weight causes the highest drop (9.5%), while half of the weight layers cause no influence on BSR. We can add those figures to our revision if needed.

As the discussion deadline is approaching, we just want to make sure that our response really makes sense to you. We are more than happy to discuss if there is still something still unclear. Can you take a look at the rebuttal and update your score if appropriate? Thank you!