Sharpness-Aware Data Poisoning Attack

We thank the reviewer for raising these concerns, and we will address them by discussing the following questions: (1) clarify the discrepancy between different attacks; (2) evaluations on the ImageNet dataset; and (3) Evaluate SAPA against SOTA defenses.

Q1: Clarify the discrepancy between different attacks

We thank the reviewer for pointing out another line of attacks that do not suffer from the re-training. We add descriptions of these attacks and clarify the discrepancy between them and our studied attacks in Appendix B.2, and refer to them in Section 2.1.

We also provide some introductions here. Some of these attacks mentioned by the reviewer, such as WaNet and LiRA, need additional assumptions that the attacker can control the re-training process. This is different from our considered "end-to-end" scenario as described in Section 2.1. Besides, BadNet attack does not generate poisoning samples targeting on specific models. Thus, it would not encounter the issue of re-training uncertainty. However, this attack typically considers inserting noticeable triggers.

Q2: Evaluations on the ImageNet dataset.

We thank the reviewer for this valuable suggestion. We conduct experiments on targeted and backdoor attacks. In detail, we test on ResNet18 and budget $\epsilon=4/255,8/255,16/255$, poison rate $p=0.1\\%, 0.05\\%$ for targeted attacks; test on ResNet18, ResNet50, VGG11, budget $\epsilon=8/255, 16/255$ and poison rate $p=0.05\\%$ for backdoor attacks. We present the success rate of Grad-Match and Grad-Match+SAPA for targeted attacks; and the success rate of Sleeper-agent and Sleeper-agent+SAPA for backdoor attacks for convenient comparison in the following Tables. We will conduct experiments on other attacks and put them in the appendix. According to the results, our method consistently improves the performance of the backbone method on the large dataset.

Q3: Evaluate SAPA against SOTA defenses.

We thank the reviewer for pointing out these defenses. We conduct experiments on targeted attacks against Deep-KNN[1] and backdoor attacks against ABL [2], Fine-pruning [3]. For both attacks, we test on CIFAR-10 with $\epsilon=16/255$ and $p=1\\%$. Results(success rate) for Deep-KNN are shown in the first Table below and results(success rate) for ABL and Fine-pruning are in the second Table below. According to the results, Deep-KNN can hardly defend our attacks and our method can consistently boost the performance of backbone methods against powerful defenses such as ABL and FP.

We hope our responses can address your concerns. We appreciate your valuable reviews and look forward to further feedback.

References

[1] Deep k-NN Defense against Clean-label Data Poisoning Attacks

[2] Anti-Backdoor Learning: Training Clean Models on Poisoned Data

[3] Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks

We thank the reviewer for raising this concern. We would like to clarify why some baselines are lower than reported in the original paper. First, as mentioned in the footnote of Table 1, the original paper of Grad-Match only trains 40 epochs while we train for 160 epochs, and we include a detailed analysis of the impact of epochs in Appendix D. Second, as we mention in the footnote of Table 2, Hidden trigger and Clean-label are originally proposed for fine-tuning settings (by fixing the early layers), while we focus on an end-to-end setting in this work, and our results are consistent with results in work [1].

We hope our responses can address your concerns. We appreciate your valuable reviews and look forward to further feedback.

[1] Sleeper agent: Scalable hidden trigger backdoors for neural networks trained from scratch.

We are grateful for your reviews. We hope that our responses have addressed your concerns. If you have any further concerns, please let us know. We are looking forward to hearing from you.

Q7: Clarification for Eq.7.

Eq.7 serves to approximate the worst-case adversarial loss utilizing the concept of sharpness with respect to a given poisoning set $D_p$and an adversary set $D_T$. We would like to point out that our poison crafting does not directly utilize Eq.7; instead, the actual optimization problem we engage with is formulated in Eq.8. This is where the bi-level optimization problem is defined, leading us to the generation of $D_p$.

As detailed in Algorithms 1-4, the initial poisoning set $D_p$ starts with a random initialization and is iteratively updated during the generation process. Thus we do not need to generate them twice. The updating of $\theta^*$ depends on the backbone methods SAPA is combined with. For SAPA+Grad-match, $\theta^*$ is the model trained on clean data and fixed during the generation which follows the original algorithm in [1]; for SAPA+Error-min and SAPA+Error-max, $\theta^*$ is also randomly initialized and iteratively updated as shown in Algorithm 1 and 4.

[1] Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching

We thank the reviewer for raising these concerns. We will address them by discussing the following questions: (1) Why does crafting on the worst-case poisoned model intuitively lead to more generalizable/potent poisons on average; (2) the reference of the term "sharpness" in this work; (3) discussion about some statements mentioned by the reviewer; (4) does our work explicitly mitigate the concern that existing attacks are not architecture-agnostic; (5) Gradient matching considers ensembles, can SAPA be combined with ensembling and restarts (6) can SAPA improve other objectives other than gradient matching; and (7) clarification for Eq.7.

Q1: Why does crafting on the worst-case poisoned model intuitively lead to more generalizable/potent poisons on average?

We would like to provide some intuition about the "worst" re-trained model below. As we mentioned in Section 4.1, the worst-case poisoned model is the model that has the worst poisoning effect among the possible models that are trained on the poisoned dataset (considering various training uncertainties including initialization, training algorithm, etc.). In other words, the poisoning effect on the worst-case model serves as the lower bound of poisoning effects for all models. Therefore, in the re-training stage, the poisoning effect of the real victim model should also be better than this worst-case model, so by maximizing the poisoning effect on the worst-case model we can guarantee that a high poisoning effect is preserved for other models.

Q2: The reference to the term "sharpness" in this work.

We thank the reviewer for pointing out the difference between the sharpness in this work and the sharpness of the "standard" loss landscape. We refine our presentation to avoid potential misunderstanding about the concept of sharpness in the revised paper. Specifically, we clarify this in the paragraph above Eq.(8) where the sharpness is defined.

Q3: Discussion about some statements mentioned by the reviewer

We agree that we should be more careful of these statements. We remove the unjustified statements such as the first one. For the second statement, our experiments can provide some empirical justifications. According to our experimental results in Table 1-3, SAPA improves the attacking performance of backbone methods, indicating that more poisoning effects are preserved. The curve of the poisoning effect in Figure 1 also provides empirical evidence that SAPA leads to better poisoning effects.

Q4: Does our work explicitly mitigate the concern that existing attacks are not architecture-agnostic?

We agree that we do not devise explicit strategies to address the uncertainty due to model architecture. Thus, we correspondingly emphasize this fact at the end of Section 4.1 in our paper.

Besides, according to our experimental results in Table 4 where poisons are generated from one model and tested on models with different architectures, our method can improve the poisoning effect of backbone methods. This result suggests that SAPA could also implicitly bring improvements under the model architecture variation.

Q5: Gradient matching considers ensembles, can SAPA be combined with ensembling and restarts?

We are aware that Gradient matching considers ensembles in the original paper. Thus, in our paper, we have also compared our method with Grad-match with ensembles in Section 5.5 and Figure 2. In detail, we compare SAPA with Grad-match incorporated with different setups of Ensembling and Re-initialization. From the result, we can find that our method can achieve comparable performance to the Grad-matching with a large number of ensembled models, while our method has a much better time efficiency. As a conclusion, we believe that time efficiency is an advantage of our method. As shown in Figure 2, the ensembled Grad-match needs much more time than our method (without ensembles) to achieve a similar performance.

We also acknowledge that SAPA can be combined with Ensembling and Re-initialization (E&R). Thus, in Appendix Table 11, we have conducted experiments to apply E&R to SAPA on both targeted and backdoor attacks. According to these results, E&R can further improve the performance of SAPA while with the cost of time efficiency.

Q6: Can SAPA improve objectives other than gradient matching?

Yes, apart from Grad-match, we also combine SAPA with objectives of un-targeted attacks, i.e. Error-min in Section 4.3 and Error-max in Appendix C.3. We conduct experiments for both objectives and show them in Table 3. We observe obvious improvement for both attacks, and this further illustrates the wide range of applicability of SAPA.

We are grateful for your reviews. We hope that our responses have addressed your concerns. If you have any further concerns, please let us know. We are looking forward to hearing from you.

We thank the reviewer for raising these concerns. We will address these concerns by discussing the following questions: (1) Why the success rate is low when the budget is low and how to improve it; (2) evaluation on more defenses; (3) why the performance against adversarial training is not good enough; and (4) additional comments.

Q1: Why the success rate is low when the budget is low?

We thank the reviewer for raising this concern and we agree that it is more difficult to conduct attacks when the budget is very low. However, a low perturbation budget could possibly make the attacks become more unnoticeable in practice. Thus, there is a chance that the attacker can correspondingly increase the poisoning rate for a better poisoning effect. As illustrated in the following Table, by increasing the poisoning rate to 5%, we observe a significant enhancement in the success rate, exceeding 30%.

Therefore, these unnoticeable attacks, though have low average success rates, can still be harmful. Moreover, existing works in poisoning attacks [10][11][12] also consider this scenario, and our work follows the same setting. The experimental results have shown that our method outperforms them, as seen in Table 1-2 in our paper.

Q2: Evaluation on more defenses

We thank the reviewer for pointing out these defenses, and we conduct experiments to evaluate our method. We test on the CIFAR-10 dataset with ResNet18. We notice that these defenses are designed for backdoor attacks, so we conduct on backdoor attacks only and fix the budget $\epsilon=16/255$ and poison rate p=1%. We report the success rate in the following Table. According to these results, our method can improve the backbone attack's resistance against various defenses.

Q3: Why the performance against adversarial training is not good enough?

We admit that the performance against adversarial training is not good enough in Table 5. In fact, adversarial training is a powerful defense against un-targeted attacks, and previous works such as [7][8][9], also suffer from this defense. According to our experiments in Table 5, our method can improve the resistance of backbone attacks and outperform all the baselines. It remains a challenge to break the defense of adversarial training and we will keep exploring in this direction.

Additional comments

We really appreciate the reviewer's suggestions. We will include evaluations on defenses in Q2, as well as image visualizations, in the appendix. We apologize for the typos in that sentence in Section 5.5, which should be 'existing attacks'. We have carefully revised the paper and ensured no other typos. We also revise Table 3 to make it more readable.

References

[1] Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks

[2] Spectral signatures in backdoor attacks

[3] Spectre: Defending against backdoor attacks using robust statistics.

[4] Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering.

[5] STRIP: A Defence Against Trojan Attacks on Deep Neural Networks

[6] Rethinking the backdoor attacks' triggers: A frequency perspective.

[7] Unlearnable Examples: Making Personal Data Unexploitable

[8] Adversarial Examples Make Strong Poisons

[9] Autoregressive Perturbations for Data Poisoning

[10] Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching

[11] MetaPoison: Practical General-purpose Clean-label Data Poisoning

[12] Poison frogs! targeted clean-label poisoning attacks on neural networks

We are grateful for your reviews. We hope that our responses have addressed your concerns. If you have any further concerns, please let us know. We are looking forward to hearing from you.

We thank the reviewer for raising these valuable comments. In rebuttal, we provide clarifications on the following issues: (1) the threat models for different attacks; (2) a general description for SAPA; and (3) compare our work with another existing work.

Q1: Clarification for the threat models.

We follow the reviewer's suggestion and add details about the threat model for each type of attack, including the attacker's goal, attacker's capacity, and adversary objectives. We put these details in Appendix B.1 and refer to them in Section 2.1.

Q2: A general description of SAPA.

We thank the reviewer for giving this suggestion. We follow the reviewer's suggestions and modify the names of methods in the experiment part.

In Eq.(8) in section 4.1, we define a general objective for SAPA, where we replace the original adversarial loss with a sharpness variant to approximate the local worst-case loss. Due to the distinctions between backbone algorithms, we also separately discuss the implementation of SAPA for different attacks, shown in Section 4.2, 4.3 and Appendix C.2 respectively.

Q3: Compare our work with another existing work.

We thank the reviewer for pointing out the recent method [1] in un-targeted attacks. According to Table 4 and Section 4.4 in the original paper [1], SEP can achieve better performance than the ensembled error-max (referred to as AdvPoison in [1]) while having better time efficiency. In our rebuttal, we also present a direct comparison of SEP, Error-max and SAPA+Error-max, Error-min and SAPA+Error-min under $\epsilon=8/255$ in the following Table. The results of SEP and SAPA are directly collected from the paper [1] and our paper respectively.

According to these results, SEP can have the best performance in the studied methods. However, we would like to emphasize that SAPA is designed as a general strategy that can enhance existing attacks. As shown in this Table, SAPA improves the performance of backbone methods and reduces the gap between them and SEP.

Moreover, we notice that it is also possible to combine SAPA and SEP for further improvement. For instance, we can replace the original loss for each checkpoint in Equation 3 in [1] with a sharpness-aware loss. We are studying this problem and we will share the result if time permits.

[1] Self-Ensemble Protection: Training Checkpoints Are Good Data Protectors

We are grateful for your reviews. We hope that our responses have addressed your concerns. If you have any further concerns, please let us know. We are looking forward to hearing from you.

We thank the reviewer for raising these concerns. We have carefully revised our paper and refined the presentation. In the rebuttal, we will provide clarifications on (1) the novelty of our work; (2) the relation between our work and the theoretical analysis in [2]; and (3) do we have experiments on the transferability of poison effects.

Q1: The novelty of our work.

We agree with the reviewer that our proposed method applies an existing algorithm in [1] to tackle the loss-landscape sharpness. However, we provide a new perspective on leveraging sharpness to solve one of the critical challenges in data poisoning attacks. In detail:

• Sharpness [1] was previously used to improve the generalization ability of models.
• We focus on the uncertainty problem of poisoning attacks during the re-training process. Different from existing poisoning attacks, we aim to generate poisoning data with maximum poisoning effect for the approximately "worst-case" model. In our paper, we find that the problem can be solved by the existing sharpness algorithm to find the approximate worst-case model under uncertainty. We believe this is a new perspective of applying sharpness to solve a new problem.

Besides, our proposed method SAPA also differs from the existing sharpness method in terms of formulation. Existing sharpness-aware methods maximize the training loss via weight perturbation to improve the generalization of models. Differently, SAPA maximizes the loss of poisoning objective (see Eq.6 and Eq.7) to find the (locally) worst-case models via weight perturbation. This differs from the existing notion of sharpness discussed in previous studies like [1].

In summary, this paper adapts an existing concept of sharpness from a new perspective for solving a totally new problem. Based on our experiments, we validate the effectiveness and efficiency of our strategy in solving our problem for various types of poisoning attacks.

Q2: The relation between our work and the theoretical analysis in [2].

We thank the reviewer for mentioning the theoretical analysis in [2]. Proposition 1 in [2] states that under some conditions such as Lipschitz continuous gradient and the alignment between gradients of training loss and adversarial loss, the victim model will converge to a stationary point of the adversarial loss. We believe that a similar analysis could be conducted on SAPA when combining SAPA and Grad-match. Compared to the original Grad-match, SAPA only replaces the model $\theta^*$ in the original adversarial loss function $Q$ with a perturbed version, $\theta^*+v$, to derive the sharpness variant $Q^S$, as detailed in Eq.10-12, while maintaining other aspects unchanged. This will not violate the original assumption in [2], such as Lipschitz continuous gradients and loss alignment. Therefore, under similar assumptions as [2], it is also potential to show that the victim model converges to a stationary point of our adversarial loss.

Besides, we would like to respectfully note that a detailed convergence analysis of SAPA within the context of gradient matching falls outside the primary scope of our current work. Our focus in this paper is on establishing a general strategy applicable across various attacking scenarios, including targeted, un-targetd and backdoor attacks. Thus, it would be difficult to theoretically study the analytical behavior of our method in each of these attacks.

Q3: Do we have experiments on the transferability of poison effects?

We would like to politely point out that we present the analysis of transferability in the main text. As shown in Table 4, we employed ResNet18 as the surrogate model to generate poisons, subsequently testing them across a variety of victim models, including MobileNetV2, VGG11, and ViT. The results demonstrate that SAPA consistently enhances the performance of underlying methods across diverse attack scenarios. This performance improvement indicates the ability of SAPA to improve the transferability of attacks.

We hope our responses address your concerns. We appreciate your reviews and look forward to further feedback.

References

[1] Sharpness-Aware Minimization for Efficiently Improving Generalization

[2] Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching

We are grateful for your reviews. We hope that our responses have addressed your concerns. If you have any further concerns, please let us know. We are looking forward to hearing from you.

We appreciate your reviews. We hope that our responses have adequately addressed your concerns. As the deadline for open discussion nears, we kindly remind you to share any additional feedback you may have. We are keen to engage in further discussion.