BAN: Detecting Backdoors Activated by Adversarial Neuron Noise

Thank you for your valuable suggestions. We address your comments as follows.

Q1. The evaluation only focuses on CNN architectures.

A1: We provide ViT results on a 12-layer Swin transformer in the following table. For BadNets and Blend, we train the backdoored network using Adam as the optimizer. For WaNet, IAD, and Bpp, we use the default setting. All models in the following table are successfully detected as backdoored models. In addition, we notice that the attack performance of WaNet, IAD, and Bpp is not as good as BadNets and Blend. We conjecture this is because of training with SGD.

Q2. The evaluation further focuses on dirty label attacks

A2: Our submitted draft did not include clean-label attacks because it has a stricter threat model, which leads to weaker attacks. Here, we provide clean-label backdoor experiments to address the reviewer's concern using a well-known clean-label attack [A]. We take default hyperparameters from the original paper. The following table demonstrates the detection and mitigation performance on 20 networks. We use one of the 20 networks for the mitigation experiment. Experimental results demonstrate that BAN is also effective against the clean-label attack.

Q3. Does it make sense to include an approach (FeatureRE) designed for all-to-one attacks in an all-to-all setting?

A3: The results of featureRE in Table 5 are to show that methods based on all-to-one settings may not work for all-to-all. Therefore, we believe it is not misleading since the featureRE relies on the assumption that backdoor attacks are always all-to-one. We will clarify this in the revised version.

Q4. Some contributions of the paper should be stated more clearly and set apart from existing approaches in literature.

A4: BTI-DBF [1] assumes the benign and backdoor features can be decoupled by a mask for benign features and (1-mask) for backdoor features. Then, they conduct trigger inversion based on the feature mask. However, we show in the following table that their mask will be a dense matrix full of ones. The feature's shape is $512\times4\times4$ (ResNet18), meaning the maximum mask norm is 8192. The reason is that the benign and backdoor features are not prominent to each other for weaker attacks, such as BadNets and Blend. In the BadNets (w/o norm), the negative feature loss is relatively larger because the (1-mask) is almost zero, and the loss is calculated by the feature $\times$ (1-mask). To solve this problem, we apply a $L_1$ maks norm to the feature mask decoupling. In addition, the decoupling performance can be further improved by adversarial neuron noise. We use the neuron noise to activate the backdoor such that the backdoor features are more prominent to the feature mask.

Q5. Table 4: What are the BA (benign accuracy) and ASR (attack success rate) of models trained only on clean data without any poisoned data samples? It would be interesting to compare the defended backdoored models to a clean model.

A5: We provide ASR on a benign model using CIFAR-10 and ResNet18 in the following table. The ASR of other attacks is omitted because attacks such as IAD train a generator for the backdoor trigger, which does not apply to a benign model. In the table, it is clear that ASR for the benign model is very low since the benign accuracy is high.

Q6. can we assume the proposed BAN method also works reliably on datasets with semantically similar classes?

A6: The classes with semantic similarities are actually widespread. The phenomenon makes classification results of these similar classes closely related. For example, Table 5 of IB-RAR [B] shows cats and dogs (CIFAR-10 classes) tend to be classified as each other under adversarial attacks. For ImageNet200, we use class n02096294 (Australian terrier) as the target class. There are 19 kinds of terriers in our ImageNet200 datasets, such as n02094433 (Yorkshire terrier) and n02095889 (Sealyham terrier).

To further support this observation, we train 10 backdoor networks using BadNets with different target classes. In the following table, BAN is contentiously effective against the backdoor regardless of semantically similar classes.

[A] Label-consistent backdoor attacks.

[B] IB-RAR: Information Bottleneck as Regularizer for Adversarial Robustness

I thank the authors for the detailed rebuttal and additional insights. All my questions and remarks were addressed. After reading all other reviews and author responses, I decided to increase my rating.

Thank you for your valuable comments. We address your comments as follows.

Q1. The motivations of high computational overhead and reliance on prominent backdoor features are not well-illustrated in the method section. Only the limitations of BTI-DBF are discussed.

A1: High computational overhead: Most existing methods conduct trigger inversion for every class to detect potential backdoors (e.g., NC, FeatureRE, or Unicorn), so detection algorithms need to be executed multiple times, each using a different class as the optimization objective. Our detection does not rely on class-wise optimization and only executes once to detect the potential backdoor, which significantly reduces the time consumption. Similarly, BTI-DBF also only executes once for detection, but BTI-DBF needs to pre-train a U-Net generator to generate trigger images, which is computationally heavier.

Prominent backdoor features: The reliance on prominent backdoor features is illustrated in Figure 3. We show that benign and backdoor features are very close for BadNets but prominent for advanced attacks, such as WaNet, IAD, and BPP. Please also refer to the analysis under Q3.

Q2. The novelty is limited. The finding on neuron noise is previously proposed, and an additional mask regularizer based on BTI-DBF is a commonly used technique, which was used in NC as well. The idea of backdoor defense is similar to ANP, with the change in learning from the noise.

A2: Our main finding is that prominent backdoor features may not be suitable for identifying input space backdoors. Based on the findings, we show the generalization shortcomings of previous approaches, and we fix current approaches by introducing the regularizer with the further help of neuron noise. We would argue that both our findings and defenses are novel because we recognized and then tried to solve common and previously unknown shortcomings from previous research. Technically, we introduced a novel feature space regularizer (Eq. (5)) combined with adversarial neuron noise, which strategically increases the feature difference and, more importantly, helps the defense generalization.

Q3. The reliance on prominent backdoor features is expected to be illustrated in detail, and the reason why the proposed method is able to overcome it with only a mask regularizer is unclear. Some case visualizations are expected.

A3: We will revise and improve the contents in Section 3.3 and Section 4.1, together with Figure 1, to better illustrate our findings on prominent backdoor features. Regarding BAN, the mask regularizer's utility is limited, and it is only effective when combined with the adversarial neuron noise.

We also provide new experimental analysis in the table below, showing that previous decoupling methods cannot easily pick up backdoor features of weak attacks, such as BadNets, as their backdoor features are not prominent. For example, when detecting without the $L_1$ regularizer (i.e., w/o norm), the negative feature loss of BadNets is high with a very large $L_1$ mask norm, while the Bpp has an even higher negative loss with a much smaller mask norm. Note that for BadNets, the negative loss is high with a high mask norm where almost all features are included. It indicates that BadNet backdoor features are less prominent than Bpp features, making it more challenging to decouple BadNets features. We will provide a discussion in the revised draft.

(The table is also presented in the global rebuttal.) A note on the table: The shape of the feature mask is $512\times4\times4$, which means the maximum of the mask $L_1$ norm is 8192.

Q4. The conclusion from Equation 7 that ignoring the second term is expected to be illustrated with evidence such as the separate loss values on it.

A4: The table above shows the $L_1$ mask norm and separated feature losses. When optimizing without the penalty of mask norm, for relatively weaker attacks (BadNet and Blend), the mask is very dense because the $L_1$ mask norm values are close to the upper bound (8192). The negative feature loss values are also large for BadNets and Blend, but this is because the negative feature mask, i.e., (1-mask), is close to zero. Almost no feature is used to get the negative feature loss, i.e., (1-mask)$\times$feature in Eq. (5) leads to the high loss value. In other words, the high negative feature loss (without the mask penalty) is not because of backdoor features on BadNets and Blend.

With the penalty of mask norm, the optimization of the feature mask focuses more on finding backdoor features rather than only increasing the proportion of benign features. In the table, when optimizing with the penalty of mask norm, BadNets and Blend provide small negative feature loss values compared to advanced attacks (WaNet, IAD, and Bpp), even with a larger (1-mask). A small mask norm means the mask provides a larger proportion on the negative mask, i.e., (1-mask). That is to say, there are fewer backdoor features, or the backdoor features are less prominent. In conclusion, the feature decoupling using the mask is not effective when the backdoor features are not prominent.

Thank you for your appreciation of our work. We address your comments as follows.

Q1. missing explanations of the performance. For example, why is featureRE in Table 3 performing better than BAN? What would be the impact of a lambda value other than 0.5? Why is the Blend attack always better detected by BTI-DBF* than by BAN?

A1: We agree that featureRE provides slightly better benign accuracy but fails to mitigate the backdoor ASR for BadNets, Blend, and Bpp, while our method is consistently effective. Regarding the difference between featureRE and BAN, we observed that commonly (e.g., BTI-DBF or BAN), including featureRE, there is a trade-off between benign accuracy and backdoor deletion, where the backdoor deletion task harms benign accuracy.

About the $\lambda$ values: We provided experiments with different values, as shown in Table 7 in the appendix (we tested 0.2, 0.5, 0.8, and 1). We saw that as the lambda value increased, the BA dropped.

For the Blend attack, as shown in Table 3, the Blend backdoor is only better detected by BTI-DBF* on ResNet18, but our method performs better on VGG16 and DenseNet121.

Q2. missing relationship between the adversarial activation noise approach and input/feature perturbation approach? How is the adversarial activation noise approach different from assuming Lipschitz continuity on the estimated input-output function?

A2: Thank you for this intriguing question. From the literature, it is known that for a backdoor attack, if we have a small trigger that changes the output of a benign input into a malicious target label, this behavior can be related to the high Lipschitz constant [A] and a neural network with high robustness then tends to have a lower local Lipschitz constant. Moreover, a larger local Lipschitz constant implies steeper output around trigger-inserted points, leading to a smaller trigger effective radius and making trigger inversion less effective [B]. Thus, the concepts of adversarial activation noise and Lipschitz continuity are related, and the local Lipschitz constant can serve as an upper bound for the trigger’s effective influence. We will include a discussion about this in the revised version of the document.

In addition, introducing theoretical tools like the Lipschitz constant for backdoor defense may also be very tricky in practice because it needs approximation for implementation. For example, [A] evaluates the channel-wise Lipschitz constant by its upper bound but does not thoroughly discuss the relationship between the channel-wise Lipschitz constant and the network-wise Lipschitz constant, where the theorem of Lipschitz continuity really relies on. In another recent paper [C], it is also mentioned that empirically estimating the Lipschitz constant is hard from observed data, which usually leads to overestimation. We admit that methods relying on Lipschitz continuity may not require heavy computational load and are also related to our approach. However, our method emphasizes more on fine-tuning with the guidance of neuron noise, rather than tuning the trained model.

[A] Zheng, R., Tang, R., Li, J., Liu, L. Data-Free Backdoor Removal Based on Channel Lipschitzness. ECCV 2022.

[B] Rui Zhu, Di Tang, Siyuan Tang, Zihao Wang, Guanhong Tao, Shiqing Ma, Xiaofeng Wang, Haixu Tang. Gradient Shaping: Enhancing Backdoor Attack Against Reverse Engineering. NDSS 2024.

[C] Data-Driven Lipschitz Continuity: A Cost-Effective Approach to Improve Adversarial Robustness

Q3. The authors state, “We also exploit the neuron noise to further design a simple yet effective defense for removing the backdoor, such that we build a complete defense framework.” The statement “we build a complete defense framework” is an overstatement in the paper.

A3: We agree with the reviewer that this claim may not be accurate in some scenarios, and we rephrase the sentence as follows to avoid confusion: "We also leverage the neuron noise to design a simple yet effective defense that removes the backdoor."

Thank you for your valuable comments. We address your comments as follows.

Q1. I might have a misunderstanding, but ... be decoupled.

A1: Our defense aims to activate the backdoor by neuron noise such that the backdoor networks behave differently from benign networks. The feature mask we use in Eq. (5) is improved by the penalty of mask norm, and it does depend on the prominent backdoor. However, we apply the feature mask to activated backdoors (networks with neuron noise), which makes the decoupling easier. The existence of backdoored neurons is validated in [A,B,C,D]. As there are both backdoor and benign neurons in backdoored networks and only benign neurons in benign networks, it is evident that they will be different when applying noise to those neurons. Note that using neuron noise is insufficient for backdoor detection, as shown in Figure 2.

Q2. Additionally, in Section 4.4, the paper claims that ... in this context

A2: The feature decoupling methods, including other methods such as BTI-DBF and featureRE, may not always be effective. However, different from other methods, we apply the feature mask to a network with neuron noise. The noise activates the potential backdoor, which makes our decoupling easier.

Q3. There is a minor issue that ... Table 3 and Table 4

A3: We agree with the reviewer that performance improvement is sometimes not substantial. Our aim is to mitigate the generalization shortcomings of previous approaches. In our experiments, BAN is the only effective approach for all types of attacks.

Q4: Is the detection performance ... backdoor?

A4: BAN is designed to be unaware of the trigger pattern, and we included backdoor attacks that are based on different patterns in experiments, including BadNets, Blend, IAD, WaNet, BppAttack, Adap-blend, and SSDT.

Regarding the backdoor training strength, we provide experimental results with different poisoning rates of BadNets in the following table since the poisoning rate is closely related to backdoor strength. We use BadNets because it is relatively weak at a low poisoning rate, while more advanced attacks may still be strong at a low poisoning rate. All models in the table are successfully detected and mitigated. Our new results indicate that a stronger backdoor is easier to detect.

Q5: I'm curious about ... makes feature decoupling less effective

A5: To validate this hypothesis, we trained a 4-layer MLP with the BadNets attack and another benign MLP for the CIFAR-10 dataset. In the table, the ``num. to target'' refers to the number of samples (in 5000 validation samples) that are classified as the backdoor target after our detection. After BAN detection, we find BadNets MLP classifies 3607 samples as the target, while for benign MLP, it is 419. It means that BAN detects the backdoor.

We also find that the positive feature loss (benign) is very close to the negative loss (potential backdoo). It indicates that the backdoor features are more challenging to decouple from benign ones, as the reviewer hypothesized. Note that the performance drop of MLP might also make features challenging to recognize.

Q6. I also believe... applying the mask (1-m)

A6: In practice, selecting the lambda value is easy. As discussed in the method section, the motivation for using the mask norm ($L_1$ norm) with lambda in Eq. (5) is to ensure the optimization objective is decoupling between benign and backdoor features. Without the constraint of the mask norm in Eq. (5), the optimization objective will be simply increasing the mask norm unless there are extremely strong backdoor features. Therefore, we choose a lambda value for Eq. (5) such that the mask norm does not change significantly while optimizing.

This selection is easy because we only need to check the value of the mask norm. For example, the following table shows the $L_1$ mask norm and positive and negative feature losses. The feature size is $512\times 4\times 4$, which means the maximum of the mask norm is 8192 (every value in the mask is one). It is clear that the mask is almost full of ones when $\lambda_1$ is smaller than 0.7, and the negative feature loss (backdoor feature) is ignored. Therefore, in this case, we need the $\lambda_1$ values to be greater than 0.7. The selection of the lambda values is unaware of potential backdoors.

[A] ABS: Scanning neural networks for back-doors by artificial brain stimulation, CCS 2019

[B] Backdoor Scanning for Deep Neural Networks through K-Arm Optimization, ICML 2021

[C] Reconstructive neuron pruning for backdoor defense, ICML 2023

[D] Pre-activation distributions expose backdoor neurons, NeurIPS 2022

Thank you for your appreciation of our work. We address your comments as follows.

Q1. A figure of the proposed method outlined in Sections 3.1 and 3.3 could be a helpful tool to visualize the proposed method.

A1: We will add a figure that illustrates the design of our method.

Q2. The in figure text in Figures 2 and 3 are too small and hard to read. Some tables use % and others don't when report accuracy, e.g. Table 2 and Table 3 with no % symbol for the BA columns.

A2: We will increase the font size and revise the table coherence.

Q3. I think a table or figure further emphasizing the proposed changes would greatly improve the strength of this paper.

A3: We will revise the details of different methods and show the differences between our method and baselines.

Q4. What is the impact of $\lambda_2$ on the fine-tuning loss found in Equation 8?

A4: The hyperparameter $\lambda_2$ controls the trade-off between benign accuracy (BA) and attack success rate (ASR). Increasing the $\lambda_2$ value will decrease BA but provide better defense performance. Table 7 shows the results with different $\lambda_2$ values, ranging from 0.2 to 1, where $\lambda_2=0.5$ provides the best defense performance.