EigenGuard: Backdoor Defense in Eigenspace

1. About the difference between our paper and Universal Perturbations:

Although trigger features can be regarded as a kind of universal perturbation that lies in eigenspace corresponding to the top singular value, the former work doesn’t show that trigger features are low rank and can only lie in top eigenspace. Furthermore, the relationship between the feature dimensions and backdoor accuracy during training as shown in Figure 1 is only unevaluatedin our paper. Therefore, the new findings of our paper are that the trigger features can only located in the top low-rank subspaces. Such a phenomenon may be caused by the low poison rate of the trigger samples.

Furthermore, we also designed our new method by damping top singular values and using features corresponding to more eigenspace for prediction. Therefore, models with our module can naturally defend against many backdoor attacks.

2. About “lack of evidence that the unlearning will lead to bad natural accuracy” and “ finetuning is better than EigenGuard”:

From Table 3, one can see that ABL’s natural performance is not good, which is caused by its unlearning loss.

As for finetuning, its natural accuracy may be better sometimes because they use additional clean datasets for finetuning while ours do not. But its backdoor accuracy is not stable which means that fine-tuning is not an effective way for backdoor defense.

3. About why modifying eigenspace will not cause the accuracy drops:

Because we do not drop features, we only rescale the singular values corresponding to each eigenspace. As our experiments show, the trigger features only lie in a low-rank space with large singular values, our method will lead neural networks to make predictions based on more features instead of trigger features. Therefore, natural features will help the neural network to predict natural labels instead of trigger labels on backdoor samples.

As for natural samples, one can notice that models with our EigenGuard do not drop any information. Therefore, the natural performance will not be affected.

4. Why not just let top singular values to zero?

Just removing the top eigenspace will make the natural accuracy drop because features corresponding to top singular values are also important for natural classification, as shown in the following experiments:

Our method tries to avoid trigger features dominating the prediction. According to our theory, if the trigger features not dominating, which means the corresponding singular value is not large, the rest natural features will help models predict other classes instead of natural classes.

5. About the 0.001 scaling factor.

This parameter will not influence much, we don’t set it to zero because we think the gradient corresponds to the latter eigenspace may also help. But in fact, such a hyper-parameter will not influence much.

6. About the theoretical proof.

Our theory illustrates that damping the dominating trigger features may make the models use natural features to make correct predictions. It is also consistent with our method, we make the top singular values smaller to avoid the triggers dominating classifications.

7. About more baselines:

As for baselines, we choose the most effective attacks and defenses as baselines according to BackdoorBench, since we use its setting for evaluation. We also add some defense methods(NAD[1], Spectral signature (SS)[2]) and attack methods (WaNet[3], SSBA[4]) on CIFAR-10. The full table is shown below:

Thanks for your review, the following are our responses.

1. About the possible adaptive attacks:

A possible adaptive attack is trying to expand the trigger feature space and make it as complicated as the natural features. Then only damping top eigenspaces for features is not useful because trigger features are varied enough and no natural features are used for classification.

However, we think it is not easy to achieve because too complicated backdoor features will also make the backdoor not easy to implement. Because the success of backdoor attacks benefits from the simple shortcuts between trigger features and labels. Therefore, the attack may be less effective when trigger features are complicated.

2. About CLB attacks and the choice of baselines.

We also evaluate the CLB attacks on CIFAR-100 and GTSRB attacks. And the results show that our methods are also effective. As for the choice of baselines, we choose the most effective attacks and defenses as baselines according to BackdoorBench, since we use its setting for evaluation. We also add some defense methods(NAD[1], Spectral Signature (SS[2]) defense ) and attack methods (WaNet[3],SSBA[4]) on CIFAR-10. The full table is shown below:

Thanks for your review, the following are our responses.

1. About our theoretical analysis:

Although our toy model is simple, our theory can clearly illustrate that damping the dominating trigger features will lead the models to use natural features to make correct predictions. It is consistent with our method, we make the top singular values smaller to avoid the triggers dominating classifications.

2. About the baselines in our paper:

As for the choice of baselines, we choose the most effective attacks and defenses as baselines according to BackdoorBench, since we use its setting for evaluation. We also add some defense methods (NAD[1] and Spectral Signature [2]) and attack methods(WaNet[3] and SSBA[4]) on CIFAR-10. The results show that our method can perform well on different attacks. The full table is shown below:

5. About A and k and $\sigma_k$.

A is just a matrix for calculating effective ranks. In our following evaluations, we change a feature tensor of (batch_size, channel, height, width) to be a (batch_size, channelheightwidth) matrix and calculate the effective ranks to explore the rank of the trigger feature space.

k here is the index, and $\sigma_k$ means the $k$-th singular value.

6. About the CLB attack.

We also evaluate the CLB attacks on CIFAR-100 and GTSRB datasets as shown in the above table. And the results show that our methods are also effective.

7. About the improvement of the natural accuracy.

I think the improvement of the natural accuracy is caused by the EigenGuard module can also regularize the neural network by damping the top singular values. Therefore, our EigenGuard can make neural networks’ activation to be well-conditioned. It can avoid overfitting in some circumstances and lead to better results.

[1] Neural Attention Distillation: Erasing Backdoor Triggers From Deep Neural Networks.

[2] Spectral signatures in backdoor attacks.

[3] WaNet -- Imperceptible Warping-Based Backdoor Attack

[4] Invisible Backdoor Attack with Sample-Specific Triggers

Apart from that, we also noticed that the spectral signature cannot perform well under the backdoor bench’s setting. The phenomenon may caused by its dataset split being parametric sensitive and the undetected few backdoor samples that are not detected can also lead to backdoor behavior in backdoor training, which also demonstrates that the detection and retraining defense methods are unstable.

3. About the novelty of our methods.

Our work is different from [1],[2]:

(1) First, the spectral tools are a widely used method in analysis. Therefore, the novelty of our work and former works like you proposed are the findings, not the analysis methods. And the findings and perspectives of former works and ours are different.

[1] analyzes the feature activation difference on natural and backdoor samples for a pre-trained model and [2] analyzes the gradient defense. However, our work mainly focuses on the relationship between the rank and spectrum of the trigger feature space with the attack success rate, shown in Table 2 and Figure 2. Then we conclude that trigger features’ rank is much smaller than natural features and the reason for backdoor behavior is attributed to triggers dominating the other features, which means their corresponding singular values are larger than others. Then we propose our defense method by averaging all useful features to lead the model to make the correct prediction.

(2) Secondly, the goals of these works are different. [1] tries to split datasets and retrain the model. Then they need a complicated multi-stage training procedure. [2] tries to defend against the backdoor attack with additional clean data.

However, our goal is more challenging. We would like to allow users to do end-to-end training without the concern of possible triggers in their datasets by just adding our module to their model.

[1] Spectral signatures in backdoor attacks. Advances in neural information processing systems, 31.

[2] Efficient Backdoor Removal Through Natural Gradient Fine-tuning.

4. About the threat model that our paper tries to address?

Our paper tries to achieve safe natural training against unauthorized backdoor datasets by involving a new module in the training models. With such a module, users can naturally train their models safely on unauthorized datasets without the need to adopt different methods for retraining, and purifying datasets which are both time-consuming and unstable.

The majority of model users who do not know backdoor attacks will just collect data and train them without purifying the data. Our methods are trying to protect this part of users by involving our module in released ResNet codes or other model codes.

Thanks for your reply. Here is our further clarification:

1. We don't think the success rate for new attacks and CLB attacks is high. The attack success rate shows that our model's prediction is worse than the random prediction, even the worst result SSBA is also around the random guess (10% for CIFAR-$10$). Therefore, we think the defense is successful. In the meantime, I think natural accuracy is also important because it means the utility of the models. For example, all defense methods show a clear drop in the CIFAR-100 dataset, which means these models are somehow useless.
2. About threat models, the defender can only modify their model architecture, they cannot modify the training code or datasets. Such a scenario is untouched but also realistic. For example, in distributed training, other participants cannot control the datasets or the training procedure. Furthermore, in most cases, many users won't purify their downloaded datasets and do complicated multi-stage training. Then adding our modules can make them free from backdoor threats. Since our module shows no clear degeneration in models' natural performance on various models, such a method can be directly used in any model's code like BatchNorm or ReLU.

6. About more evaluations:

We also add some evaluations for our methods, we list some results below and you may find more results on the revision with SSBA[2], WaNet[3] attack and NAD[4], Spectral Signature defense using backdoor bench[6]’s implementation on CIFAR-$10$ dataset due to the time limits. The results show that our method can perform well on different attacks.

Apart from that, we also noticed that the spectral signature cannot perform well under the backdoor bench’s setting. The phenomenon may caused by its dataset split being parametric sensitive and the undetected few backdoor samples that are not detected can also lead to backdoor behavior in backdoor training, which also demonstrates that the detection and retraining defense methods are unstable.

[1] Spectral signatures in backdoor attacks. NeurIPS, 2018.

[2] Invisible Backdoor Attack with Sample-Specific Triggers ICCV 2021

[3] WaNet -- Imperceptible Warping-Based Backdoor Attack ICLR 2021

[4] Neural Attention Distillation: Erasing Backdoor Triggers From Deep Neural NetworksICLR 2021

Thanks for your review, the following is our response.

1. About the claims in Table 1

We need to claim that our method tries to train the neural networks in an end-to-end procedure. It is a challenging setting. The goal of our paper is to allow users to do end-to-end training without the concern of possible triggers in their datasets by just adding our module to their model.

Although some methods like [1] which split the datasets and then naturally retrain the neural networks will also not influence the natural accuracy much, it is not a natural training way for neural networks. Such methods usually need a few different training stages, which is still a little complicated. Thereby, many users may still use the end-to-end training procedure instead of methods like [1]. To make it more clear, we also rewrite the tables.

2. About “However, these explanations are just a restatement of the actual method used in the paper, not based on step-by-step intuitive reasonings. I highly encourage the authors to re-write these statements and try not to rely on the observations in the figures but on the intuitions and explain why this method should work.”

Our method is proposed based on our findings on the features’ eigenspace during training on different samples and tries to propose a new module to avoid such behaviors.

First, we find that the trigger features can only concentrated in a low-rank subspace according to Figure 1 and Table 2. Then we find that trigger features are mainly distributed in the top singular space in Figure 2, which means the backdoor predictions are made because trigger features contribute a lot to the prediction due to their large singular value.

Due to these findings, we propose our EigenGuard which damps the magnitude of trigger features while involving more natural features for prediction. Then we can defend against several backdoor attacks without any additional treatments during training. It’s a natural finding to solution way for research.

3.About the “inconsistency“ between Figure 2 and our method:

Here we only damp top eigenvalues instead of dropping them because $\sigma_k$ is much smaller than $\sigma_1$, which is consistent with our findings in Figure 2. From Figure 2(a), one can see that the trigger features are large enough to dominate the classification of backdoor samples, so making it smaller can decrease its impact. In other words, we try to average the backdoor features in Figure 2 (a) and (b) to make the backdoor samples hard to classify.

The reason why we not simply removing the top eigenvalues is such a method will make natural classes hard to classify as we can see in Figure 2(b) and lead to a huge drop in natural accuracy as listed in the following experiments on CIFAR-10 with BadNets:

Also, as shown in Figure 4, the EigenGuard module can make the features perform as we wish.

4. About our methods’ relationship with Spectral Signature and Local intrinsic dimensionality.

There are two differences between ours and theirs:

(1) Our method is a plug-in module for neural architectures, neural networks can naturally defend against some backdoor attacks with end-to-end natural training by inserting our EigenGuard model into the model’s architecture. The other two methods only try to use the poisoned model's behavior to detect the trigger images and then try to purify the datasets to prevent backdoor attacks, which need complicated post-stage retraining stages.

(2) Although both works analyze the feature behaviors, ours tries to detect the feature’s spectral influence during training while others only analyze the model's feature behavior after training.

5. About the theory: Our theory tries to demonstrate that damping the dominating trigger features will make the models use natural features to make correct predictions. It is also consistent with our method, we make the top singular values smaller to avoid the triggers dominating classifications.