Fast Adversarial Training against Sparse Attacks Requires Loss Smoothing

We appreciate your constructive comments. We provide the responses to your concerns in a point-to-point manner.

1. Could authors provide some empirical results of |delta1-delta2| among different norms to support their claim?

We showcase the average $l_2$ distance between perturbations generated by 1-step and multi-step attacks $\|\delta_{one}-\delta_{multi}\|$ among different norms in the table below. The setting is the same as that in Table 5. It can be observed that $\|\delta_{one}-\delta_{multi}\|$ of $l_0$ attacks is significantly larger than that of other attacks. This is consistent with the theoretical conclusion.

In addition, we showcased the average distances between gradients induced by 1-step and multi-step attacks in Table 5 in our manuscript. As presented in Table 5, the average distance between gradients induced by 1-step and multi-step $l_0$ attacks is 5 orders of magnitude greater than those in other cases, even when a single pixel is perturbed. This further demonstrates that the optimization is challenging in the context of $l_0$ adversarial training.

2. It would be interesting to examine the performance of Expectation over Transformation (EOT) [1] and Backward Pass Differentiable Approximation (BPDA) [1].

We would like to recall that our method is built on the adversarial training framework, which is the only valid method studied in [1] to withstand adaptive attacks. As for the two mentioned methods, EOT is not expected to work since we do not randomly transform the input in evaluation, and BPDA is not applicable since non-differentiable operations, like purification and transformation, are not used in our models. Moreover, Sparse-AutoAttack, which is adopted in evaluation, is an ensemble of both white-box and black-box attacks, where gradient masking can be evaded.

3. Recent advances in addressing CO by preventing loss landscape distortion with other norms [2,3,4,5] may also prove beneficial in alleviating CO in the l0 norm and should be discussed.

We report the results of [2,3,5] in the table below. Note that we exclude the results of [4] since it utilizes interpolation operation, which assumes the adversarial budget is a convex set, while the $l_0$ adversarial budget is non-convex. The results indicate [2,3,5] experience catastrophic overfitting, thereby invalid in the $l_0$ case.

Reference

[1] Athalye, A., Carlini, N., & Wagner, D. (2018, July). Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International conference on machine learning (pp. 274-283). PMLR.

[2] Grabinski, J., Jung, S., Keuper, J., & Keuper, M. (2022, October). Frequencylowcut pooling-plug and play against catastrophic overfitting. In European Conference on Computer Vision (pp. 36-57). Cham: Springer Nature Switzerland.

[3] Lin, R., Yu, C., & Liu, T. (2024). Eliminating catastrophic overfitting via abnormal adversarial examples regularization. Advances in Neural Information Processing Systems, 36.

[4] Rocamora, E. A., Liu, F., Chrysos, G., Olmos, P. M., & Cevher, V. Efficient local linearity regularization to overcome catastrophic overfitting. In The Twelfth International Conference on Learning Representations.

[5] Lin, R., Yu, C., Han, B., Su, H., & Liu, T. Layer-Aware Analysis of Catastrophic Overfitting: Revealing the Pseudo-Robust Shortcut Dependency. In Forty-first International Conference on Machine Learning.

We are glad that most of your concerns have been addressed. Regarding your remaining question, due to the time limit, we cannot conduct a hyperparameter search for all baselines in Table 6. Nevertheless, we test different hyperparameters of two representative baselines, GA [1] and NuAT [2]. Specifically, GA is the CO defense without soft labels and trade-off loss function, and NuAT performs the best among evaluated baselines.

The results below indicate that (1) different $\lambda$ does not make GA applicable in fast $l_0$ adversarial training; (2) NuAT is sensitive to different hyperparameter choices as the training becomes unstable when $\lambda \geq 6$, and still underperforms our method, which achieves a robust accuracy of 63.0%.

In summary, this ablation study further demonstrates the effectiveness of the proposed method. If our responses address your concerns, we would be grateful if you could adjust your scores.

Reference

[1] Andriushchenko et al., Understanding and improving fast adversarial training. NeurIPS 2020.

[2] Sriramanan et al, Towards efficient and effective adversarial training. NeurIPS 2021.

We appreciate your constructive comments. We provide the responses to your concerns in a point-to-point manner.

1. The studied problem, catastrophic overfitting in $l_0$ fast AT, is quite narrow.

We would like to highlight that (1) the non-convexity of $l_0$ adversarial budget and the craggy loss landscape make fast $l_0$ adversarial training more challenging than other cases; (2) Various existing methods to mitigate catastrophic overfitting do not work in the $l_0$ case; (3) Sparse perturbations are common in physical work, so investigating robustness against sparse perturbations is important. (4) Fast $l_0$ adversarial training solves a discrete optimization problem, which is fundamentally different from the convex and continuous constraints in other $l_p$ norm cases, our methods can help other problems in this field that involve discrete optimization as well.

In the following, we elaborate on the aforementioned viewpoints:

1. Fast $l_0$ adversarial training is more challenging than the $l_1$, $l_2$ and $l_\infty$ cases. To substantiate this claim, we analyze the phenomenon of catastrophic overfitting in fast $l_0$​ adversarial training in Section 3, concluding that it arises from suboptimal perturbation locations and a significantly more challenging loss landscape compared to other norms.
2. The results in Table 6 reveal that the existing fast adversarial training methods against $l_1$, $l_2$, $l_\infty$ attacks, like random start [1], GradAlign [2], and ATTA [3], cannot avoid catastrophic overfitting in the $l_0$ scenario. Therefore, it is necessary to develop an effective fast adversarial training approach against sparse attacks to fill the void.
3. In contrast to $l_\infty$ or $l_2$ bounded perturbations, $l_0$ bounded perturbations are sparse and quite common in physical scenarios, including broken pixels in LED screens to fool object detection models and adversarial stickers on road signs to make an auto-driving system fail [6-9]. In this regard, training a robust model against l0 bounded perturbations is essential to realistic scenarios with high-security needs, such as auto-driving system and access control system.
4. Due to the unique properties of the $l_0$, we are solving an optimization problem with discrete constraints in this work. This is fundamentally different from other $l_p$ norm cases, which are continuous and convex. In this regard, the analyses and the methods in our work can help design efficient algorithms to solve other problems in this field which involve discrete optimization, such as adversarial perturbations in texts [10] and quantized parameters [11].

2. The analytical framework and the findings are similar to the existing works of analyzing overfitting in AT.

We acknowledge the relevance of AdvLC to this work and will cite it in the revised version. However, we find:

1. Although the degraded attack effectiveness can be mitigated by multi-step attacks, Figure 3 illustrates that the craggy loss landscape in $l_0$ adversarial training can still lead to catastrophic overfitting (CO). In contrast, AdvLC attributes CO to degraded perturbations and claims that the craggy loss landscape aggravates the degradation.
2. We propose theoretically guaranteed methods to improve both Lipschitz continuity and Lipschitz smoothness. However, AdvLC penalizes logit difference to regularize gradient to improve attack effectiveness, which is similar to the effect of soft labels and can only improve Lipschitz continuity.
3. We also employ AdvLC in fast $l_0$ adversarial training and obtain a robust accuracy of 47.6% against Sparse-AutoAttack, while our method achieves a robust accuracy of 63.0%. This indicates that only improving Lipschitz continuity is insufficient in the $l_0$ case, underscoring the requirement of more comprehensive smoothing techniques for $l_0$ adversarial training.

3. The mitigation methods, soft label and trade-off loss function, are proposed by the exiting works, so the methodological contribution of this work is not something novel like proposing a new method.

The differences between our work and AdvLC are elaborated in the response to the second comment.

As for the novelty, We would like to emphasize that the rationale and advantages of employing soft labels and the trade-off loss function differ significantly in the context of $l_0$ norms. Specifically, the use of soft labels and trade-off loss functions is intended to improve generalization and to balance the robustness-utility trade-off in scenarios involving other norms such as $l_1$​, $l_2$, and $l_\infty$​. However, these techniques are crucial for achieving meaningful performance in fast adversarial training under $l_0$​ constraints. Without them, fast adversarial training against $l_0$​-bounded perturbations usually suffer from catastrophic overfitting, leading to only trivial performance outcomes (see Fig. 1). In contrast, fast adversarial training using other norms can still achieve competitive results through various techniques, such as random start [1], GradAlign [2], and ATTA [3]. These methods, however, are ineffective in the $l_0$​ scenario (see Table 6). In summary, extensive analyses and experiments indicate that these smoothing techniques are the key to solving catastrophic overfitting in $l_0$ adversarial training, which is not the case for other norms.

To substantiate our claims, we analyze the phenomenon of catastrophic overfitting in fast $l_0$​ adversarial training in Section 3, concluding that it arises from suboptimal perturbation locations and a significantly more challenging loss landscape compared to other norms. In Section 4, we propose that soft labels and the trade-off loss function can provably enhance first- and second-order smoothness, respectively. Numerical results support this assertion (see Fig. 6).

In Section 5.1, we evaluate various combinations of methods that incorporate soft labels and the trade-off loss function. Our findings indicate that the integration of TRADES, SAT, and N-FGSM achieves the highest performance. Notably, our method attains the best robust accuracy of 63.0%, whereas other smoothing techniques, such as NuAT [4] and MART [5], report robust accuracies of only 51.9% and 48.0%, respectively (see Table 6).

In summary, our work is definitely not a simple extension of existing techniques to a related task. Instead, we exploit the unique challenges and solutions to a new problem not explored before. Our proposed methods borrow some elements from existing methods, but the rationale and benefits of employing them are different. Extensive experiments demonstrate that our method can reliably solve the challenge and achieve state-of-the-art performance.

Reference

[1] Wong et al., Fast is better than free: Revisiting adversarial training. ICLR 2020.

[2] Andriushchenko et al., Understanding and improving fast adversarial training. NeurIPS 2020.

[3] Zheng et al., Efficient Adversarial Training with Transferable Adversarial Examples. CVPR 2020.

[4] Sriramanan et a.l, Towards efficient and effective adversarial training. NeurIPS 2021.

[5] Wang et al., Improving adversarial robustness requires revisiting misclassified examples. ICLR 2020.

[6] Papernot et al., Practical black-box attacks against machine learning. ACM ASIACCS 2017

[7] Akhtar et al., Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access 2018.

[8] Xu et al., Adversarial attacks and defenses in images, graphs and text: A review. International Journal of Automation and Computing 2019.

[9] Feng et al., Graphite: Generating automatic physical examples for machine-learning attacks on computer vision systems. EuroS&P 2022.

[10] Yuan, Lifan, et al. "Bridge the Gap Between CV and NLP! A Gradient-based Textual Adversarial Attack Framework." Findings of the Association for Computational Linguistics: ACL 2023. 2023.

[11] Yang, Yulong, et al. "Quantization aware attack: Enhancing transferable adversarial attacks by model quantization." IEEE Transactions on Information Forensics and Security (2024).

Thanks for your constructive comments. We provide further responses to your remaining concerns:

1. Broader impact

To explore the broader impact of our method, we employ our method in adversarial training on tabular data. We aim to perturb one input feature of DANet [1] trained on the forest cover type dataset (54 features in total). The training attack is 1-step sPGD ($\epsilon_{train} = 6$) and the evaluation attack is 10000-step sPGD ($\epsilon = 1$). Note that we adapt sPGD to accommodate tabular data. The results indicate that our method is still effective for tabular data. Due to the time limit, we cannot explore more application scenarios. Despite that, the results in Table 4, 7,8,10, and below can already demonstrate the effectiveness of our method across different datasets, networks, and modalities.

2. Novelty

It is important to note that the rationale and benefits of the adopted methods are distinct from their original purpose. Specifically, SAT was proposed to improve the generalization across various tasks, while TRADES was initially designed to improve the robustness-accuracy trade-off. However, they turn out indispensable in mitigating catastrophic overfitting in fast $l_0$ adversarial training. More details are illustrated in the first point of our general response.

Similarly, the differences between FGSM [2], RS-FGSM [3] and N-FGSM [4] are that RS-FGSM adopts random initialization, which has been widely adopted in almost all aspects of deep learning; N-FGSM augments clean samples with random noise, which is generally used to improve generalization. It should be noted that neither RS-FGSM nor N-FGSM introduces any “novel” approach. However, both of them do have profound implications for fast adversarial training.

Reference

[1] Chen et al. DANETs: Deep Abstract Networks for Tabular Data Classification and Regression. AAAI 2022.

[2] Goodfellow et al. Explaining and harnessing adversarial examples. International Conference on Learning Representations (ICLR), 2015.

[3] Wong et al. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations (ICLR), 2020.

[4] Jorge et al. Make Some Noise: Reliable and Efficient Single-Step Adversarial Training. NeurIPS 2022.

We appreciate your constructive comments. We provide the responses to your concerns in a point-to-point manner.

1. The novelty is somewhat limited. The overall method may appear as a straightforward integration of existing techniques.

We would like to emphasize that the rationale and advantages of employing soft labels and the trade-off loss function differ significantly in the context of $l_0$ norms. Specifically, the use of soft labels and trade-off loss functions is intended to improve generalization and to balance the robustness-utility trade-off in scenarios involving other norms such as $l_1$​, $l_2$, and $l_\infty$​. However, these techniques are crucial for achieving meaningful performance in fast adversarial training under $l_0$​ constraints. Without them, fast adversarial training against $l_0$​-bounded perturbations usually suffer from catastrophic overfitting, leading to only trivial performance outcomes (see Fig. 1). In contrast, fast adversarial training using other norms can still achieve competitive results through various techniques, such as random start [1], GradAlign [2], and ATTA [3]. These methods, however, are ineffective in the $l_0$​ scenario (see Table 6).

To substantiate our claims, we analyze the phenomenon of catastrophic overfitting in fast $l_0$​ adversarial training in Section 3, concluding that it arises from suboptimal perturbation locations and a significantly more complex loss landscape compared to other norms. In Section 4, we propose that soft labels and the trade-off loss function can provably enhance first- and second-order smoothness, respectively. Numerical results support this assertion (see Fig. 6).

Motivated by these analyses and observations, in Section 5.1, we evaluate various combinations of methods that incorporate soft labels and the trade-off loss function. Our findings indicate that the integration of TRADES, SAT, and N-FGSM achieves the highest performance. Notably, our method attains the best robust accuracy of 63.0%, whereas other smoothing techniques, such as NuAT [4] and MART [5], report robust accuracies of only 51.9% and 48.0%, respectively (see Table 6).

In summary, our work is definitely not a simple extension of existing techniques to a related task. Instead, we exploit the unique challenges and solutions to a new problem not explored before. Our proposed methods borrow some elements from existing methods, but the rationale and benefits of employing them are different. Extensive experiments demonstrate that our method can reliably solve the challenge and achieve state-of-the-art performance.

2. The application scenario is not clearly defined. Can this adversarial training effectively enhance defense against one-pixel attacks?

In contrast to $l_\infty$ or $l_2$ bounded perturbations, $l_0$ bounded perturbations are sparse and quite common in physical scenarios, including broken pixels in LED screens to fool object detection models and adversarial stickers on road signs to make an auto-driving system fail [6-10]. In this regard, training a robust model against l0 bounded perturbations is essential to realistic scenarios with high-security needs, such as auto-driving system and access control system.

Although $l_0$ bounded perturbations have been studied in many existing works [6-10], most of them focus on generating adversarial perturbations instead of the defense against these attacks. Moreover, fast adversarial training against l0 bounded perturbations is challenging due to the suboptimality of one-step attacks and the craggy loss landscape induced by l0 bounded perturbations. What’s worse, the existing fast adversarial training methods against $l_1$, $l_2$, $l_\infty$ attacks, like random start [1], GradAlign [2], and ATTA [3], cannot avoid catastrophic overfitting in the $l_0$ scenario. Therefore, it is necessary to develop an effective fast adversarial training approach against sparse attacks to fill the void.

Additionally, our method can enhance defense against one-pixel attack by setting $\epsilon=1$. The results are reported in the table below, where vanilla denotes the model without adversarial training. The robust accuracy against one-pixel attack [15] increases from 79.9% to 81.5%.

3. The author should show the performance of these methods under l0 constraints, such as FAST-GA, FAST-BAT, FAST-PCO.

We report the results of FAST-GA and FAST-BAT below. Specifically, Fast-GA was already evaluated in our manuscript (see GA in Table 6) and was shown invalid in the $l_0$ case, the model still suffers from catastrophic overfitting. Additionally, Fast-BAT turns out unstable and performs poorly in the $l_0$ case. Note that the result of FAST-PCO is not included since it was designed for $l_\infty$ attack and utilizes interpolation, which assumes the adversarial budget is a convex set, while the $l_0$ adversarial budget is non-convex.

4. Can Fast-LS-l0 be adapted for other sparse attacks, such as those bounded by $l_1$ ?

First, we would like to point out that perturbations bounded by $l_1$ norm are not necessarily sparse [11]. Actually, [11] shows that models suffering from catastrophic overfitting in the $l_1$ norm case are usually not robust to non-sparse perturbations in the $l_1$ adversarial budget. Despite this, we still report the results on AutoAttack-$l_1$ [12].

We need to emphasize that Fast-LS-$l_0$ is not expected to be robust against $l_1$ attack since $l_1$ attack does not guarantee sparsity, because shreds of evidence suggest that models trained against specific perturbations are not robust to other perturbations [12, 13]. Nevertheless, Fast-LS-$l_0$ still significantly outperforms sTRADES, revealing its effectiveness in improving generalization among different attacks. Notably, efficient training algorithms exist to obtain competitive performance in robustness against $l_1$ bounded perturbations, such as Fast-EG-$l_1$ [11].

5. Does the trade-off loss function parameter alpha need to be adjusted across different datasets and tasks?

The corresponding ablation study was already conducted, please see the results in Table 14 in our manuscript. Specifically, the optimal $\beta$ in TRADES (i.e., $\alpha$ in trade-off loss function) is 6, which is the same as the default one that is obtained based $l_\infty$ attack. Empirically, this choice can generate competitive performance across different scenarios, which frees the practitioners of hyper-parameter tuning.

6.Could you further elaborate on the necessity of adversarial training under the l0 constraint?

It is necessary to investigate adversarial training under $l_0$ constraint. Adversarial training is shown as the most reliable method to achieve adversarial robustness in extensive case studies in [14]. Further details are in the response to the second comment.

Reference

[1] Wong et al., Fast is better than free: Revisiting adversarial training. ICLR 2020.

[2] Andriushchenko et al., Understanding and improving fast adversarial training. NeurIPS 2020.

[3] Zheng et al., Efficient Adversarial Training with Transferable Adversarial Examples. CVPR 2020.

[4] Sriramanan et a.l, Towards efficient and effective adversarial training. NeurIPS 2021.

[5] Wang et al., Improving adversarial robustness requires revisiting misclassified examples. ICLR 2020.

[6] Papernot et al., Practical black-box attacks against machine learning. ACM ASIACCS 2017

[7] Akhtar et al., Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access 2018.

[8] Xu et al., Adversarial attacks and defenses in images, graphs and text: A review. International Journal of Automation and Computing 2019. [9] Feng et al., Graphite: Generating automatic physical examples for machine-learning attacks on computer vision systems. EuroS&P 2022.

[11] Jiang et al., Towards Stable and Efficient Adversarial Training against l1 Bounded Adversarial Attacks. ICML 2023.

[10] Wei et al., Unified adversarial patch for cross-modal attacks in the physical world. ICCV

[12] Croce et al., Mind the Box: l1-APGD for Sparse Adversarial Attacks on Image Classifiers. ICML 2021.

[13] Zhong et al., Towards Efficient Training and Evaluation of Robust Models against l0 Bounded Adversarial Perturbations. ICML 2024.

[14] Athalye et al. "Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples." ICML 2018.

[15] Tom Yuviler and Dana Drachsler-Cohen. 2023. One Pixel Adversarial Attacks via Sketched Programs. Proc. ACM Program. Lang. 7, PLDI, Article 187 (June 2023).

It is a gentle reminder that the discussion period is coming to an end. We kindly request you to review our response to see if it sufficiently addresses your concerns. Your feedback is of great importance to us. For your convenience, we summarize some important points during the discussion with other reviewers here:

1. In the responses to reviewer wG55, we further evaluate our method on tabular data. The results indicate that our method is still effective for tabular data. The results in Table 4, 7,8,10, and there demonstrate the effectiveness of our method across different datasets, networks, and modalities.

2. We further highlight that the rationale and benefits of the adopted methods are distinct from their original purpose, and they turn out indispensable in mitigating catastrophic overfitting in fast $l_0$ adversarial training. More details are illustrated in the second point of the responses to reviewer wG55.

3. Following the suggestion of reviewer vSUq, we conduct additional experiments to show that the poor performance of evaluated baselines is not attributed to the hyperparameter selection. This further indicates the effectiveness of our method.

3. Additional Experiments

To address the reviews’ concerns, we further compare our method with other methods to mitigate catastrophic overfitting. The results listed below suggest that our method achieves the strongest robustness against sparse attacks. More specific experimental results are included in the point-to-point responses.

4. Revision to the Manuscript

We revise the manuscript as follows and highlight them in red.

• Cite and discuss [10-14] in the manuscript.
• Add results of suggested methods [10-14] to mitigate catastrophic overfitting in Table 6 of the manuscript.

Reference

[1] Wong et al., Fast is better than free: Revisiting adversarial training. ICLR 2020.

[2] Andriushchenko et al., Understanding and improving fast adversarial training. NeurIPS 2020.

[3] Zheng et al., Efficient Adversarial Training with Transferable Adversarial Examples. CVPR 2020.

[4] Sriramanan et a.l, Towards efficient and effective adversarial training. NeurIPS 2021.

[5] Wang et al., Improving adversarial robustness requires revisiting misclassified examples. ICLR 2020.

[6] Papernot et al., Practical black-box attacks against machine learning. ACM ASIACCS 2017

[7] Akhtar et al., Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access 2018.

[8] Xu et al., Adversarial attacks and defenses in images, graphs and text: A review. International Journal of Automation and Computing 2019.

[9] Feng et al., Graphite: Generating automatic physical examples for machine-learning attacks on computer vision systems. EuroS&P 2022.

[10] Zhang Y. et.al “Revisiting and advancing fast adversarial training through the lens of bi-level optimization” PMLR, 2022

[11] Li, Lin, and Michael Spratling. "Understanding and combating robust overfitting via input loss landscape analysis and regularization." Pattern Recognition 136 (2023): 109229.

[12] Grabinski, J., Jung, S., Keuper, J., & Keuper, M. (2022, October). Frequencylowcut pooling-plug and play against catastrophic overfitting. In European Conference on Computer Vision (pp. 36-57). Cham: Springer Nature Switzerland.

[13] Lin, R., Yu, C., & Liu, T. (2024). Eliminating catastrophic overfitting via abnormal adversarial examples regularization. Advances in Neural Information Processing Systems, 36.

[14] Lin, R., Yu, C., Han, B., Su, H., & Liu, T. Layer-Aware Analysis of Catastrophic Overfitting: Revealing the Pseudo-Robust Shortcut Dependency. In Forty-first International Conference on Machine Learning.