Democratic Training Against Universal Adversarial Perturbations

Response to questions:

1. Thanks for the comments. Indeed, our initial observation was based on DF-UAP (as described in Section 3.2). However, after we evaluated our approach against multiple UAP attack methods besides DF-UAP, i.e., sPGD, LaVAN and GAP, it becomes apparent that our method indeed works for all the UAP attacks that we tested. In fact, we further evaluated one recent UAP attack SGA [4] and the result of SGA is shown below:

Moreover, we analyse the layer-wise entropy on clean samples and samples with UAP on original model and defended model. Comparing the entropy spectrum of clean samples and UAP perturbed samples on the original model, sPGD, LaVAN, GAP and SGA will cause layer-wise entropy to drop. While for UAP perturbed samples on Democratic Training enhanced model looks similar to that of clean samples, which suggest that our defense is able to mitigate the effect of different types of UAPs on entropy spectrum. We will include the entropy plots in the revised version.

2. Thanks for the comments, $H(i)$ represents the layer-wise entropy described in equation (3) for sample $i$. We will improve our presentation in the revised version. Different from generating a UAP, SG generates low entropy samples for model finetuning. Since we focus on targeted UAP attacks, methods relying on generated UAPs often require generating UAPs of a set of target classes when the attack target is unknown. However, SampleGenerator does not need target class information. Furthermore, we compare the effectiveness of adversarial examples and low entropy samples generated by SampleGenerator when finetuning a given model in RQ3. The results show that, with the same finetuning process setting, adversarial examples are less effective compared with low entropy samples in mitigating the effect of UAPs. Hence, based on above results, low entropy samples generated by our SampleGenerator are more powerful compared to adversarial examples and UAPs, which does not require any information on the attack target class.

References

[1] Costa, J. C., Roxo, T., Proença, H., & Inácio, P. R. (2024). How deep learning sees the world: A survey on adversarial attacks & defenses. IEEE Access.

[4] Liu, Xuannan, et al. "Enhancing generalization of universal adversarial perturbation through gradient aggregation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

We appreciate your thoughtful review and the constructive feedback provided. Below are our responses to the points you've raised.

Response to weaknesses:

1. Although it is right to say that our method is based on min-max, we would respectfully argue that our observation and method are still novel - as it is based on entry spectrum rather than adversarial attacks. That is, based on our observation that UAP causes abnormal entropy, Democratic Training mitigates the effect of UAPs in general by finetuning a given model with low entropy samples generated on-the-fly. Different from adversarial training which trains a model from scratch, Democratic Training improves the robustness of a given model against UAP attacks more efficiently. Based on our experimental results in RQ3, our low entropy samples are more effective compared with adversarial examples when finetune a given model against UAP attacks. To further compare Democratic Training and adversarial training, we evaluate TRADES[1] which is a widely recognized adversarial training method on UAP defense and the results are shown below:

Based on the above result, both TRADES and Democratic Training are effective in mitigating the effect of UAPs. However, TRADES sacrifices model accuracy for over 10% while Democratic Training keeps the model accuracy high (reduced by 3%). Furthermore, Democratic Training repairs the model within 20min while it takes more than 15hrs for TRADES to train a robust model on the same machine. Hence, adversarial training is effective in keeping models robust against UAP attacks but not time efficient. Democratic training finetunes a given model for a few epochs which is much faster and is able to reduce the attack success rate effectively while keeping the model accuracy on clean samples at a high level

2. Yes, $L_cce$ represents cross entropy loss and $I_b^{en}$ represents a batch of generated low entropy samples. Thanks for pointing this out and we will improve our presentation in the revised version.

3. Thanks for the suggestion and the effort in trying out our approach. We tested our approach on a wideresnet model training on CIFAR10 and below table shows the result. We are glad and preparing to open source our code (https://anonymous.4open.science/r/democratic_training-EB5A/ ).

Most of the conclusions in the author's paper and response come from experimental results, and I am willing to believe in the author's experimental results, so my problem has been solved and I will improve my score. But considering that I haven't seen the author's code and detailed experimental setup, my repeated experiments on other datasets are not satisfactory, so I have to lower my confidence.

Response to questions:

1. In RQ3, we aim to compare the efficiency of adversarial examples and our low-entropy examples in mitigating the effect of UAPs when finetuning a given model. Moreover, we conduct an additional experiment to evaluate adversarial training from scratch over UAP attacks. We compare the UAP defense performance of TRADES[1] (which is widely recognized adversarial training method) and ours on a wideresent model trained on cifar-10 dataset. The result is summarized below:

Based on the above result, both TRADES and Democratic Training are effective in mitigating the effect of UAPs. However, TRADES sacrifices model accuracy for over 10% while Democratic Training suffers from much less reduction (3%). Furthermore, Democratic Training repairs the model within 20 min while TRADES takes over 15 hrs to train a robust model on the same machine. Hence, our method is effective in UAP defense and is much more time efficient.

2. Thanks for the comments. In this work, we would like to use entropy to characterise how uncertain the model is on the classification of the intermediate features. Higher entropy suggests the features are ambiguous while lower entropy indicates the model is more certain on classifying the features. Based on our empirical study, UAPs will cause the layer-wise entropy to drop, and such lower entropy indicates the model is more certain on its classification at the same layer. Existing work [12] shows that UAPs contain dominant features over original image and we argue that such dominant features cause the layer-wise entropy to drop which dominates the model prediction. We will improve our presentation accordingly in the revised version.

References

[1] Costa, J. C., Roxo, T., Proença, H., & Inácio, P. R. (2024). How deep learning sees the world: A survey on adversarial attacks & defenses. IEEE Access.

[12] Zhang, Chaoning, et al. "Understanding adversarial examples from the mutual influence of images and perturbations." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020

2. Thanks for the insightful comments. In this work, we focus on targeted UAP attacks which are both more relevant from an attacker point of view (i.e., so that the attacker can trigger specific target outcomes) and more challenging from a defender point of view. Our empirical study in section 3.2 shows the abnormal entropy spectrum caused by targeted UAPs and Democratic Training is designed accordingly. We have now managed to evaluate Democratic Training on non-targeted attacks as well and the results are summarized below: | | before ||| after ||| | ----- | ----- | ----- | ----- | ----- | ----- | ----- | | model | cacc | aacc | SR | cacc | aacc | SR | | NN1 | 0.752 | 0.057 | 0.939 | 0.705 | 0.594 | 0.267 | | NN2 | 0.717 | 0.056 | 0.943 | 0.651 | 0.369 | 0.559 | | NN3 | 0.685 | 0.098 | 0.888 | 0.65 | 0.469 | 0.408 | | NN4 | 0.999 | 0.002 | 0.981 | 0.968 | 0.918 | 0.066 | | NN5 | 0.858 | 0.053 | 0.958 | 0.839 | 0.607 | 0.374 | | NN6 | 0.892 | 0.289 | 0.737 | 0.884 | 0.801 | 0.129 | | avg | 0.817 | 0.093 | 0.908 | 0.783 | 0.626 | 0.301 |

As we can see from the above table, although not designed for non-targeted UAPS, Democratic Training still reduces the attack SR from over 90% to 30% on average. This is indeed not as effective as targeted UAP defense performance and we believe this is due to the different entropy spectrum caused by the two types of UAPs. We also analyzed the entropy spectrum of clean and UAP perturbed samples and no clear separation of the two is observed (we will add the plot to the revised version). Hence, although non-targeted UAPs does not cause severe entropy change, enhancing a given model with low-entropy samples still improves the robustness against such perturbations to a certain level.

3. Thanks for the comments and the details of each network in Table 4 is shown below:

For comparison with existing works. We compare the performance of CFN and FNS against Democratic Training on NN1, NN2 and NN3. Instead of testing different combinations of clr and dr setting, we adopt the recommended value in the original paper. The table below shows the details for each model. For SFR, as the method is not fully open-sourced, we can only evaluate the performance of GoogleNet trained on ImageNet dataset where a pretrained defense model is provided (Table 5 shows the result on GoogleNet).

Thank you for your thorough review and valuable insights. We have provided detailed responses to your comments below. Response to weaknesses:

1. Thanks for the comments and we are glad to evaluate more recent UAP attacks. For the time being, we evaluated Democratic Training against SGA [4] published in 2023 and below shows the result: | | before || after || | ------- | ----- | ----- | ----- | ----- | | model | Aacc | SR | Aacc | SR | | NN1 | 0.133 | 0.722 | 0.592 | 0.005 | | NN2 | 0.067 | 0.806 | 0.415 | 0.096 | | NN3 | 0.147 | 0.641 | 0.510 | 0.011 | | NN4 | 0.034 | 0.999 | 0.904 | 0.009 | | NN5 | 0.107 | 0.798 | 0.743 | 0.020 | | NN6 | 0.227 | 0.776 | 0.812 | 0.034 | | average | 0.119 | 0.790 | 0.663 | 0.029 |

The results show that Democratic Training is able to reduce the attack success rate from 81% to 2.9% and model accuracy is maintained high (>71%), which is consistent with the results reported in the draft. We will add these new results.

Response to questions:

1. We are glad to extend our method on ViT and other neural network architectures in our future works. Sorry to say that due to the time constraint, we haven’t been able to finish experiments on transformers yet.

2. Sorry for omitting the information. The time cost of enhancing all models are listed below (which we will add in the draft).

References

[5] Shen, Guangyu, et al. "Backdoor scanning for deep neural networks through k-arm optimization." International Conference on Machine Learning. PMLR, 2021.

[6] Tang, Di, et al. "Demon in the variant: Statistical analysis of {DNNs} for robust backdoor contamination detection." 30th USENIX Security Symposium (USENIX Security 21). 2021.

[7] Liu, Yingqi, et al. "Complex backdoor detection by symmetric feature differencing." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[8] Wu, Dongxian, and Yisen Wang. "Adversarial neuron pruning purifies backdoored deep models." Advances in Neural Information Processing Systems 34 (2021): 16913-16925.

[9] Ho, Chih-Hui, and Nuno Vasconcelos. "DISCO: Adversarial defense with local implicit functions." Advances in Neural Information Processing Systems 35 (2022): 23818-23837.

[10] Akhtar, Naveed, Jian Liu, and Ajmal Mian. "Defense against universal adversarial perturbations." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[11] Borkar, Tejas, Felix Heide, and Lina Karam. "Defending against universal attacks through selective feature regeneration." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

We greatly appreciate your comprehensive review and thoughtful suggestions. Please find our detailed responses to your comments below. Response to weekness:

1. Thanks for the comments and we have evaluated Democratic Training on adaptative attacks. The results are summarized below and are also available in Appendix 7.4 ADAPTIVE ATTACKS . In summary, although secondary UAP attacks on Democratic Training repaired models can still generate UAPs that successfully fool the models, our defense keeps the secondary attack success rate to a very low level while keeping the adversarial accuracy high.

Furthermore, we consider more advanced adversaries who may tailor the adversarial examples trying to bypass democratic training. To explore Democratic Training’s resilience to such attackers, we conduct experiments such that when generating UAP, the attacker further control the change in layer-wise entropy. Based on DF-UAP, the optimisation loss function is modified as $L(i) = (1 - weight) * L_{cce}(i, target) - weight * H_l(i)$, where $i$ represents a clean training inputs, target represents the attack target class and $H_l(i)$ represents the layer-wise entropy loss. We use $H_l(i)$ to control the entropy change caused by the UAP and parameter $weight$ is used to control the importance of $H_l(i)$ over attack success rate. We conduct such advanced attack on all models with $weight$ set to 0.1~0.9 and and all models how similar results. For illustration purpose, results on $NN_1$ are summarised below:

Based on above result, increasing en_weight will cause the attack performance to drop, i.e., the attack success rate starts to drop when $weight$ > 0.5 and the attack SR is below 60% when en_weight is set to 0.9. Our defense stays effective across different en_weight settings where the attack SR is recued to <1% for all scenarios. We observe similar results on other models as well. Hence, knowing how Democratic Training enhance the model and control the change in layer-wise entropy during attack process, the adversary is still not able to bypass our defense effectively.

2. Thanks for the recommendation and we are glad to extend our work to more types of networks in our future work. Sorry to say that due to the time constraint, we haven’t been able to finish experiments on transformers yet.

3. Thanks for the comments and we agree that it might now always be the case that a clean dataset is available. In our problem definition, our aim is to protect a model trained by a third party. In this scenario, a small set of clean data is usually available for testing and validation purposes. We believe such assumption is reasonable in practice and is usually made in existing works on defense against neural network attacks [5,6,7,8,9,10,11]. Having said that, we are glad to explore data-free defense methods against UAPs in our future works (for instance, using a synthetic clean dataset).

Thanks for the response. It solves my concerns. I'd like to update the score.

4. Thanks for the comment. We are glad to evaluate Democratic Training on smaller-scale datasets. Below is the performance on a wideresenet model rained on cifar-10. It can be observed that the results are consistent with what we reported. We will add the details in the draft.

5. Thanks for the comments, we will improve our background section in the revised version. The three existing methods we are comparing with are designed for UAP defenses. We are glad to extend the comparison against more general adversarial defenses. Given the time constraint, we have now managed to compare with two additional two defense methods. That is, we evaluated TRADES against UAPs and compared the result with Democratic Training. Furthermore, we evaluated the performance of DensePure [3] on the wideresenet trained on cifar-10 the the performance comparison is shown below:

DensePure is effective in reducing the UAP attack success rate, but model accuracy is reduced by 12%. Democratic Training reduces the attack success rate to 3.4% and model accuracy is maintained high (3% reduction). Furthermore, as DensePure is a technique based on input sample purification, a large overhead is incurred for each inference. Based on our experiment, DensePure introduces about 560s overhead per inference (with the recommended setting).

6. We will fix these issues in the revised version accordingly.

References

[1] Costa, J. C., Roxo, T., Proença, H., & Inácio, P. R. (2024). How deep learning sees the world: A survey on adversarial attacks & defenses. IEEE Access.

[2] Zhang, Hongyang, et al. "Theoretically principled trade-off between robustness and accuracy." International conference on machine learning. PMLR, 2019.

[3] Chen, Zhongzhu, et al. "DensePure: Understanding Diffusion Models towards Adversarial Robustness." Workshop on Trustworthy and Socially Responsible Machine Learning, NeurIPS 2022. 2022.

We deeply appreciate your thorough evaluation and insightful feedback. Below, we provide our detailed responses to the points raised. Response to weaknesses:

1. Thanks for the comments and sorry for the mistake. Indeed, our approach is designed to defend against a strong adversary who can conduct UAP attacks with white-box access to the model. We have modified our threat model in the revised paper accordingly.
2. Thanks for the insightful comment. In RQ3, we aim to compare the efficiency of adversarial examples and our low-entropy examples in mitigating the effect of UAPs when finetuning a given model. We are happy to evaluate ‘full’ adversarial training on UAPs and compare the result with ours. Given the time constraints, we have now evaluated TRADES [2] on a wideresent model trained on cifar-10 dataset and compared the result with ours. The results are summarised below.

Based on the above result, both TRADES and Democratic Training are effective in mitigating the effect of UAPs. However, TRADES sacrifices model accuracy for over 10% while Democratic Training suffers from much less reduction (3%). Furthermore, Democratic Training repairs the model within 20 min while TRADES takes over 15 hrs to train a robust model on the same machine. Hence, our method is effective in UAP defense and is much more time efficient.

3. Thanks for the recommendation and we agree that advanced adversaries may tailor the adversarial examples trying to bypass democratic training. To explore Democratic Training’s resilience to such attackers, we conduct experiments such that when generating UAP, the attacker further control the change in layer-wise entropy. Based on DF-UAP, the optimisation loss function is modified as $L(i) = (1 - weight) * L_{cce}(i, target) - weight * H_l(i)$, where $i$ represents a clean training inputs, target represents the attack target class and $H_l(i)$ represents the layer-wise entropy loss. We use $H_l(i)$ to control the entropy change caused by the UAP and parameter $weight$ is used to control the importance of $H_l(i)$ over attack success rate. We conduct such advanced attack on all models with $weight$ set to 0.1~0.9 and and all models how similar results. For illustration purpose, results on $NN_1$ are summarised below:

Based on above result, increasing en_weight will cause the attack performance to drop, i.e., the attack success rate starts to drop when $weight$ > 0.5 and the attack SR is below 60% when en_weight is set to 0.9. Our defense stays effective across different en_weight settings where the attack SR is recued to <1% for all scenarios. We observe similar results on other models as well. Hence, knowing how Democratic Training enhance the model and control the change in layer-wise entropy during attack process, the adversary is still not able to bypass our defense effectively.

Dear reviewer xSaF:

Thanks for your comments and we will add these changes in our revised paper. If there are any specific areas where additional detail or explanation could further enhance the presentation, we would be happy to address them.

Response to additional points:

Comparison with TRADES:

1. Thanks for the positive assessment of our results and we appreciate your recognition of its overall quality.
2. Sorry about the mistake, and yes previously we report the advacc after defense for our method and TRADES. Below is the result on both Advacc and SR.

3. Thanks for the recommendation and the machine we used to run all reported experiments is with 96-Core 1.4GHz CPU and 60GB system memory with an NVIDIA 24GB RTX 4090 GPU.
4. Thanks for the recommendation and we are glad to combine Democratic Training with adversarial training like TRADES. This may further improve the performance of Democratic Training and would likely to help extending Democratic Training to other types of adversarial attacks besides UAPs.

Adaptive Attack

Thanks for the comments and below table summarize the mean entropy of samples with and without UAPs generated with different $weight$ settings:

*clean sample entropy is 7.1

As we can see from above result, as $weight$ increases, the entropy value of samples with UAP also increases and the value exceeds that of clean sample when $weight >= 0.6$. It is also observed that the attack success rate starts to drop when $weight$ is greater than 0.6. With our enhanced model, the entropy value of samples with UAP stays high for different $weight$ settings and the attack success rate is kept below 1%. These results show that even when the advanced attacker tries to control the entropy drop when training an UAP, Democratic Training is still able to protect a given model effectively.