Breaking the False Sense of Security in Backdoor Defense through Re-Activation Attack

We deeply appreciate the reviewer's valuable time and thoughtful comments.

Q1. Details of the surrogate-based attack (i.e., the transfer attack), and the pros and cons of the surrogate-based attack in comparison with the black-box attack (BBA).

R1: Thanks for this constructive comment. We would like to refer you to our response to Q2 of Reviewer G6gk, where we clarify the threat model, main attack steps, as well as additional evaluations and analyses.

We compare our transfer attack (TA) and BBA in two aspects:

• Attack effectiveness: According to reported results in Table 2 and Table 3 of the main manuscript, TA achieves higher ASR than BBA in most cases because:
  • For BBA: The effectiveness of BBA is limited by the specific black-box attack method (Square attack in this work). More advanced method might improve the performance. It is also limited by query numbers, where we show that ASR can be improved with increasing query numbers in Table 2 of Common Responce.
  • For TA: As analyzed in the Common Responce, there is a backdoor effect similarity between the original backdoored model and the defense model. Thus it is easier to optimize a highly transferable trigger.
• Attack efficiency: BBA requires lots of queries to the target model to achieve satisifying attack performance, while TA just needs one query to attack the target model. Obviously, TA is more efficient and more practical than BBA.

In summary, TA has better performance on both effectiveness and efficiency than BBA. However, we would like to emphasize that our main goal of designing re-activation attack algorithms under three scenarios is to verify the backdoor existence in defense models. All three attacks could be further improved by developing more advanced algorithms.

Q2. Detectability of the attacked poisoned instances (i.e., new poisoned samples obtained by our re-activation attack) is not discussed.

R2: Thanks for this constructive comment. It is valuable to study whether the proposed attack will increase the detectability during inference. Our study is as follows:

• Settings: We adopt three representative inference-time poisoned sample detection methods including SCALE-UP [1], SentiNet [2], and STRIP [3]. The detection task requires two input arguments, including the model and the query datasets. We evaluate on five pairs, including <the orginal backdoored model $f_{\text{A}}$, the original poisoned dataset $\mathcal{D}_p$>, <the defense model with FT-SAM defense $f_{\text{D,FT-SAM}}$, dataset $\mathcal{D}_p$>, <model $f_{\text{D,FT-SAM}}$, the re-activation dataset $\mathcal{D}_{p,\Delta{\xi}}$>, <the defense model with SAU defense $f_{\text{D,FT-SAM}}$, dataset $\mathcal{D}_p$>, <model $f_{\text{D,SAU}}$, the re-activation dataset $\mathcal{D}_{p,\Delta{\xi}}$>.
• Results and analyses:
  • For test-time detection, the result in Table 1 shows our attacks do not markedly increase the TPR compared the the other two pairs. More detection performance on our BBA and TA are shown in Table 6 of PDF. The appearance of generated poisoned samples are also provided in Figure 2 of PDF.
  • For test-time defense, our attack under test-time defenses has been shown in the response to Q4 of Reviewer AhAB, where our attack maintains a certain ASR against these defenses.

In conclusion, these findings offer valuable insights, paving the way for the future design of more stealthy re-activation attacks.

Table 1: Detection performance (TPR %) of different poisoned samples under three detection methods.

Q3. Only mean results are shown.

R3: Thanks.

• Results: Here we provide the mean results with standard deviations (std.) under five different random seeds among several attacks and defenses in Table 2 for our black-box attack. More results of our WBA and TA is presented in Table 4 and 5 in PDF.
• Analysis: It can be observed that the std. is small among these attack and defense method across three of our attacks. It shows our attack methods are stable.

Table 2: Mean result with std. for our BBA.

Q4. Description of the running time.

R4: Thanks.

• We would like to clarify that the running time of our attack is only related to trainning dataset and network, while independent with specific backdoor attack or defense methods. Thus, we didn't specify the particular method, as the running time is consistent across methods.
• Regarding the "N/A" in Table 10, we executed WBA and TA on the CLIP models. For query-based black-box attack, the attacker cannot directly access the target model (such as weights or gradients) and CLIP models only return the final matching score or ranking results. This limits the ability of query-based black-box attacks. Moreover, there are no relevant studies for reference. Thus, we marked related result as "N/A".

We hope this explanation resolves your queries. We welcome your feedback or further inquiries. Thanks again.

[1] SCALE-UP: An efficient black-box input-level backdoor detection via analyzing scaled prediction consistency, ICLR2023.

[2] Sentinet: Detecting localized universal attacks against deep learning systems, SPW2020.

[3] STRIP: A defence against trojan attacks on deep neural networks, ACSAC2019.

We sincerely thank the reviewer for the valuable time in reading our work and constructive concerns. We are encouraged by the positive comments of very interesting insights and clear description.

Q1. Whether backdoor re-activation is creating a novel backdoor or not.

R1: Thank you for your insightful question. We would like to refer you to the common response for a comprehensive analysis on the relationship between the original backdoor attack (OBA) and our re-activation attack (RBA) in three perspective.

Moreover, we want to clarify that the attacks proposed in this work involve three scenarios, namely white-box scenario, black-box scenario, and transfer-based scenario, where the latter two representing more realistic situations. However, none of the three attacks involve revising model parameters. Particularly, black-box attacks and transfer attacks operate under the assumption that the attacker lacks access to the target model's parameters. Thus, we are not engaged in any form of weight manipulation.

Overall, our re-activation backdoor attack shows a strong correlation with the original backdoor and our attacks involve no modification of network weights. Hope the above points could address your questions.

Q2. Clarification of the setting of transfer attack (TA).

R2: Thanks for this concern. We would like to clarify the setting of transfer-based re-activation attack (TA), and present some analyses about its effectiveness.

• Setting of TA in our main manuscript.

  • Threat model: As demonstrated in Line 132-135 of the main manuscript, the adversary trains a backdoored model $f_\text{A}$ and releases it to users. The user receives model $f_\text{A}$, and obtains a defense model $f_\text{D}$ based on $f_\text{A}$ by some post-training defense. Thus, the adversary does not know the exact defense method, but has full information about the original trigger $\boldsymbol{\xi}$ and model $f_\text{A}$ which has same model architecture as $f_\text{D}$.
  • Main steps in TA: There are three main steps in TA. Step 1: Based on $f_\text{A}$, the adversary can obtain a surrogate model (or a collection of surrogate models) by utilizing some existing post-training backdoor defenses, denoted as $f_{\text{D}'}$. Step 2: The adversary conduct a white-box re-activation atack against $f_{\text{D}'}$, based on the original trigger $\xi$, to obtain a new trigger $\xi_{D'}'$. Step 3: The adversary adopts $\xi_{D'}'$ to attack the target defense model $f_{\text{D}}$ during inference.
  • Why TA is effective for re-activating backdoor: As analyzed in Q1 of Common Response and Sec. 4.4 in the main manuscript, the original backdoor still exists in defense models. In other words, the backdoor effects of both the surrogate model $\xi_{D'}'$ and the target model $\xi_{D}'$ are highly similar with that of the original backdoored model $f_\text{A}$. Thus, it is natural to deduce that $\xi_{D'}'$ and $\xi_{D}'$ also have similar backdoor effects. It explains the high ASR of our TA.

• Further evaluation with a more strict setting: transfer attack across model architectures.

  • Threat model: Here we present a more strict setting that the adversary can only manipulate training dataset, while has no access to the training and post-training stage. Thus, the adversary only knows the original trigger $\boldsymbol{\xi}$, but has no knowledge of $f_\text{A}$ or $f_\text{D}$. Compared to above threat model, one major challenge is the unknown architecture of the target model $f_\text{D}$.
  • Main attack steps: Compared to the steps in the above setting, there is one additional step that the adversary should firstly train a backdoored model $f_\text{A}'$ based on $D_p$, which has different architectures with $f_{\text{D}}$. All remaining steps are same with those in the above setting.
  • Experimental results: As shown in Table 1, although transfer attack across model architectures doesn't achieve as high ASR as the transfer attack with the same architecture (i.e., results in Table 3 of the main manuscript), it still has a certain degree of transferability of backdoors, which is an intriguing phenomenon worthy of further exploration.

In summary, the effectiveness of the transfer re-activation is mainly due to the high backdoor effect similarity between surrogate and target defense models, even in different architectures. We think its performance could be further improved by borrowing ideas from the field of transfer adversarial attacks. Its practical threat should be seriously considered by future defenses.

Table 1: Transfer re-activation attack ( ASR %) against the target model PreAct-ResNet18, using different source models (WideResNet28-2, ResNet18, VGG19-BN).

Q3. Paper structure can be improved. R3: Thanks for this constructive suggestion. We would like to clarify that since the backdoor existence phenomenon and its metric BEC are the most critical contributions of this work, we emphasize them in both Abstract and Introduction. Following your suggestion, we will expand Sec. 3.2 by adding some analysis about the characteristics of the proposed BEC score in the revised manuscript.

Finally, we sincerely hope that above responses could well address your concerns. We will be more than glad to have further discussions with you if there is any remaining concerns. Once again, we express our heartfelt gratitude for your valuable and constructive comments.

Firstly, we would like to show our gratitude for your valuable time and professional reviews, as well as the recognition of our efforts. We wish to further clarify and summarize our analyses to address your remaining concern.

• The original backdoor and the re-activated backdoor are not exactly same. But, they are highly similar on backdoor effect, close to each other, and highly different with another new backdoor (i.e., the general universal adversarial perturbation attack). Since the parameter weights of the original backdoored model and the defense model are different, and their corresponding triggers are different, thus we say they are not exactly different. For the second sentence, we believe the global response has shown very clear evidences. In short, from the intrinsic perspective, our claim of re-activate the original backdoor, rather than creating another new backdoor is correct.
• In terms of transfer attack, in the previous responses, we have shown that the surrogate models with same architectures or different architectures with the original backdoored model are effective in transfer attack. The reason is that both the surrogate model and the backdoored model are trained on the same poisoned dataset, which is the origin of the backdoor, thus, they are highly similar with each other on backdoor effect. Hence, the backdoor re-activation is independent with the choice of surrogate. We didn't see any conflict with the phenomenon of backdoor re-activation. If you can further specify your point, that will be very helpful.

We really hope above clarification could address your remaining concern. We would like to further discuss with you, and the deep discussion with you are very valuable to reveal the real value of our work.

We really understand your caution when facing a new finding. As said by Albert Einstein,

"The framing of a problem is often far more essential than its solution".

We strongly believe that the problem of backdoor re-activation observed in this work could significantly influence the development of backdoor learning, to force people to reconsider the effectiveness of post-training backdoor defense, which is now one of the main streaming types of backdoor defenses.

We are lucky that your rigorous and professional reviews are very critical to help us to give a very solid definition and analysis about this new problem. We sincerely appreciate that you can take more time to consider our response, and give a comprehensive assessment of the value of this work.

I too think that the phrasing such as "backdoor re-activation" and "dormant backdoor" can be confusing/misleading because it suggests that the model has some state that is changed by "re-activation" (that "re-activation" changes the model parameters). However, the model parameters remain the same since the attacker cannot influence them after training. What changes by "re-activation" is the state of the attack, based finding a different backdoor that is already present in the defended model.

I suggest a slightly different framing that might hopefully be more clear and universal.

A backdoor is a property of the model instance and the goal of the attacker. If there exist a (universal) adversarial perturbation that can cause the model instance to behave as desired by the attacker, the model has a backdoor, and the (universal) adversarial perturbation is the corresponding trigger.

The backdoor is a natural backdoor if it is present independently of whether the training data is poisoned.

The goal of a defense is to prevent/remove the backdoor (ideally all backdoors) compared to an undefended model.

However, as demonstrated in the paper, many defenses remove the backdoor initially intended by the attacker, but there remains a different backdoor that is stronger than a natural backdoor, and the methods presented in the paper can find it.

What do you think?

We greatly appreciate your patient explanation, and now we clearly get the point that makes you confusing.

Firstly, we think the descriptions from Paragraph 3 to 5 (i.e., "The backdoor is a property ... compared to an undefended model.") are very very clear, showing your deep understanding of backdoor learning.

However, in terms of the description "but there remains a different backdoor" in Paragraph 6, we would like to further discuss with you. Let's consider backdoor learning from the adversary's perspective.

• The adversary's ultimate goal is to cause a model to behave as it desires during inference, i.e., predicting the poisoned sample that contains a trigger as the target label, while predicting clean samples correctly.
• To ahieve that goal, usually, the adversary should do two steps:
  • Step 1: Backdoor injection. It aims to form a stable mapping from a trigger to the target label in a model, through manipulating the training dataset (i.e., data poisoning based), manipulating the training process (i.e., training controllable based), or manipulating both.
  • Step 2: Backdoor activation. When the backdoored model is deployed by the model owner to provide query service to the customer, the adversary will try to activate the injected backdoor to achieve its goal, via querying the model with the poisoned sample containing a trigger.
• What will happen if the backdoor activation fails? There will be two results for the backdoor activation step, i.e., success or fail. If success, then the adversary wins. If fail, what the adversary can do? Firstly, it may guess that some post-training defenses have been conducted by the owner on the backdoored model to change the parameters, such that the original mapping from the original trigger to the target label cannot be activated. Unfortunately, there is no chance for the adversary to change the model. If the adversary aims to re-activate the backdoor, its only choice is changing the trigger, using our re-activation attack methods. Although the new trigger is different with the trigger used in backdoor injection, and the model is also changed by the defender, the ultimate goal of the adversary has been achieved.
• For backdoor attack, is it necessary to keep the trigger same in both backdoor injection and backdoor activation? Our answer is NO. We notice that, in most existing backdoor attacks, there is one default setting that, the triggers used in backdoor injection and backdoor activation are same. But, is it really necessary? NO.
  • Firstly, from the above description of the adversary's ultimate goal (or from previous backdoor learning works), we don't see such a requirement. Thus, changing this setting will not violate the definition of backdoor attack.
  • Actually, this setting has been violated in a few existing works, such as Alpha-Blend [1] (where it calls asymmetric trigger), and distributed Backdoor attack [2]. However, note that their activated model is still the original backdoored model, and their succeeding reasons are mainly due to the trigger generalization phenomenon. They are intrinsically different with our re-activation attack, as the model has been changed by the defender, and the succeeding reason is also different (please refer to the common response, here we don't repeat).

In summary, from the adversary's perspective, we think it is reasonable to call our attack as backdoor re-activation, because its goal has been achieved, though the trigger is not exactly same with the trigger used in backdoor injection and the backdoored model is also changed by the defender.

We sincerely hope the above descriptions can make the terminology more clear. However, we will not insist on what it calls. The most important thing is to make real contribution to the field of backdoor learning. We believe, and you may have agreed with us that, this work could attract the researchers' attention to re-investigate the effectiveness of post-training backdoor defenses, and inspire future works for safer defenses.

The in-depth discussions with you are very enjoyful, and really helpful to make this work more solid. Greatly appreciated.

References:

[1] Revisiting the Assumption of Latent Separability for Backdoor Defenses. ICLR, 2023.

[2] Distributed Backdoor Attacks against Federated Learning. ICLR, 2019.

Dear Reviewer G6gk,

We sincerely hope that our latest response to your remaining concern, as well as the discussions with Reviewer AhAB under your concern, could be helpful to address your concern. Looking forward to your feedback, and we are glad to discuss with you for any remaining concern.

We sincerely appreciate your valuable time and professional comments.

Best regards, Authors

Thank you for the positive review on the interesting observation. Your insightful questions and concerns are greatly appreciated. Please inform us if these responses effectively address all your inquiries.

Q1. The differences among our re-activation attack (RBA), the original backdoor attack (OBA), and general universal adversarial perturbation attack (gUAA, i.e., natural backdoor).

R1: Thanks for your insightful concerns. In the Common Response, we conduct a comprehensive comparison among our RBA, the original backdoor attack (OBA), and gUAA. This analysis is undertaken through three distinct aspects: the activation mechanism of the backdoor effect, the search rate, and the robustness against random noise. The three-perspective analyses verify that our RBA method finds a highly correlated backdoor with the original backdoor, rather than a less correlated backdoor (i.e., new backdoor), or a general universal adversarial perturbation (i.e., natural backdoor). Thus, we can claim that our RBA actually re-activates the orginal backdoor.

Please refer to Common Response for our analysis. Hope above points could address your concern, and we will clearly clarify the differences in the revised manuscript.

Q2. Suggestion of adding evaluations of some recent defenses, such as SEAM and CT.

R2: Thanks for this constructive suggestion.

• Additional evaluations: Following your suggestion, we evaluate the performance of our re-activation attack against SEAM and CT, respectively. The evaluations are conducted on CIFAR-10 dataset with PreAct-ResNet18 network, and the results are shown in Table 1. It is found that both SEAM and CT are vulnerable to the proposed re-activation attack.
• Clarification: We would like to emphasize that we have never claimed that all post-training defenses are vulnerable to re-activation attack, which is not rigorous. The main aim and contribution of our work are (1) revealing this new treat, which has been verified on several classic post-training defenses, and (2) providing effective tools to evaluate the vulnerability of any old or new post-training defenses. Consequently, the future post-training defenses should consider this threat and bypass the proposed re-activation attack.

Table 1: ASR (%) of our RBA attack against SEAM and CT.

Q3. What if the original trigger is not used and the perturbation is simply optimized?

R3: Thanks. Our response is expanded from two aspects:

• If the original trigger is not used, it devolves into a general universal adversarial perturbation attack (gUAA). Actually, the distinctions between our RBA attack (starting from the original trigger) and gUAA has been carefully analyzed from three perspectives in the Common Response. It is clear that our re-activation attack re-activates the orginal backdoor attack, rather than finds a general universal adversarial perturbation or a natural backdoor.
• Additional comparions under the white-box attack setting and analysis. In addition to the black-box comparions shown in Table 2 of the Common Response, here we also supplement the white-box comparisons, as shown in Table 2 below. Although the ASR values of gUAA is also high, its activation mechanism is intrinsically different with our re-activation attack (refer to the analysis in Common Response). gUAA just verifies a well-known threat once again that DNN models are likely to be vulnerable to (universal) adversarial attacks. In contrast, our re-activation reveals a new threat to existing post-training backdoor defenses.

Table 2: Comprisons between our RBA and gUAA under the white-box setting, measured by ASR (%).

Dear Reviewer ULSi,

We sincerely wish to convey our gratitude for your investment of time and insightful remarks. Your forthcoming feedback is greatly anticipated, specifically regarding the issues we've tackled in our rebuttal. Our primary aim is to assure that our rebuttal is closely aligned with your suggestions. Your contributions are important to the enhancement of our work.

Best regards, Authors

I appreciate the authors' comprehensive rebuttal. It addresses most of the concerns, and I am raising my score. Please include all the results and discussions in the final version. Thanks!

Firstly, we would like to show our sincere appreciation to the reviewer for dedicating the valuable time, offering gracious affirmation, and providing constructive.

Q1. Why is CKA a good choice in Eq. (2) and (3)?

R1: Thanks for this insightful comment. We would like to explain from two aspects:

• Our task and challenge: In this work, we need to measure the quantity of backdoor existence within a model by measuring the backdoor effect similarity between models. We calculate the similarity between feature maps in backdoor related neurons, which are structured and high-dimensioned feature representations. For such representations, traditional metrics such as $L_k$ norm distance or cosine distance could disrupt the structure information of these representations.
• CKA is highly suitable for our task: CKA [1] measures the similarity between two representations, which utilizes HSIC to measure the independence between two distributions. CKA is widely used for measuring similarity of high dimensional representations between networks and thus is highly suitable for our task. Its effectiveness is verified in our work by comparing the relationship between BEC and our re-activation attack ASRs (Fig.3(b)).

Q2. Why does BEC include intermediate model activations in Eq. (3)?

R2: We compute BEC across the entire network because backdoor attack has a cumulative effect, and each layer contributes to a successful backdoor attack. We have empirically demonstrated that our BEC metric and the re-activation ASR have a high positive correlation in Fig. 3(b) in the main manuscript and the details of ASR with BEC are shown in Table 1. The positive correlation shows that BEC is effective in measuring backdoor existence.

Table 1: Comparison between ASR and BEC.

Q3. Why is $\lambda$ set to 1 rather than 0? What if $\lambda=0$?

R3:

• The function: The second term in Eq.(4) serves as a regularization item, aiming to encourage the model to minimize the highest probability of non-target classes, thereby enhancing the potency of the attack. If $\lambda=0$, it might lead to the model maintain high output probabilities for non-target classes, thus potentially diminishing the attack performance.
• Results: Table 2 illustrates the results of WBA when $\lambda=0$. It shows a slight decrease in the overall attack effectiveness with $\lambda=0$.

Table 2: Re-activation attack performance (%) under different values of $\lambda$.

Q4. Attack against test-time defenses.

R4: Thanks. Here we test our attack against test-time defenses.

• Settings: With the poisoned samples optimized by our re-activation attack on defense models with FT-SAM, we test the effectiveness against test-time defenses STRIP[2], ZIP[3], and SCALE-UP[4].
• Results: Table 3 shows that our attack maintains a high ASR against ZIP. For the SCALE-UP and STRIP, there is a significant decrease in ASR. However, the ACC of the model is low, meanwhile.
• Analysis: This experiment inspires future design of attack methods capable of evading test-time defense. Strategies might include methods to closely match the feature distributions of clean data, and preventing overly strong activations.

Table 3: Performance (%) against test-time defenses.

Q5. The SIG example in Fig. 5.

R5: Thanks. It’s somewhat an illusion. The mechanism in use is actually an additive trigger. This aligns accurately with the approach described in the original paper. For the crop, we will modify it in the revision.

Q6. Some unclear phrases.

R6: Thanks for your careful checking of our manuscript. Our clarification of the indicated phrases is as follows:

• L9: "lie dormant". It means that the backdoor cannot be activated by the original trigger, but it still exists in the model. L163: What is BEC of $f_{\theta_A}$?It is 1,because $\rho_{\text{BEC}}$ ($f_{\theta_{\text{A}}}$, $f_{\theta_{\text{A}}}$,$f_{\theta_{\text{C}}}$;$D_p$) = $\frac{1}{N}$ $\sum_{l=1}^{N}$ $\frac{S_{\text{A},\text{A}}^{(l)}(D_{p})-S_{\text{C},\text{A}}^{(l)}(D_{p})}{S_{\text{A},\text{A}}^{(l)}(D_{p})-S_{\text{C},\text{A}}^{(l)}(D_{p})}=1$.

Note that the second and third arguments in $\rho_{\text{BEC}}$ serves as two reference models to measure the backdoor existence of the model corresponding to the first argument.

• L168: "greater existence". We will modify it to stronger existence.
• L242: What do the percentages represent in particular? They represent absolue improvements on ASR values.
• L243: The claim (2) is not completely accurate. Thanks for this careful comment. Following your suggestion, we will revise it to "which shows the effectiveness of our re-activation attack method" in the manuscript.

Q7. Writing mistakes and suggestions.

R7: We appreciate your attentiveness to the errors in our manuscript. We will revise the paper based on your suggestions.

[1] Similarity of Neural Network Representations Revisited, ICML2019.

[2] STRIP: a defence against trojan attacks on deep neural networks, ACSAC2019.

[3] Black-box backdoor defense via zero-shot image purification, NeurIPS2023.

[4] SCALE-UP: an efficient black-box input-level backdoor detection via analyzing scaled prediction consistency, ICLR2023.