On the Onset of Robust Overfitting in Adversarial Training

We thank the reviewer for the insightful comments.

Q1: Undefined words and lack evidence to support the claims.
A1: The non-effective features denote the robust features in the training set that lack generalization. The term "learning state gap" between the training and test sets has been revised to "robustness gap" in our updated manuscript. This change has been made consistently throughout the paper. Additionally, we employ quantitative metrics such as adversarial loss and robust accuracy to measure the model's robustness, providing evidential support for the existence of a robustness gap between the training and test datasets, as illustrated in Figure 1(b). Regarding the generation of non-effective features, our analysis is as follows: during the adversarial training process, the model's adversarial robustness on the training set exceeds that on the test set, as evidenced by the adversarial loss and robust accuracy observed in Figure 1(b). This results in a robustness gap between the training and test data. Considering that adversarial perturbations are generated on-the-fly and adaptively adjusted based on the model's robustness. The robustness gap between the training and test data leads to distinct adversarial perturbation on the robust features in these datasets. The varying degree of adversarial perturbation on the robust features in the training and test data amplify the distribution differences between these two datasets, thereby degrading the generalization of robust features on the training set and facilitating the generation of non-effective features. The increasing non-effective features further exacerbates the robustness gap between training and test data, forming a vicious cycle. These analyses are detailed in our updated manuscript presented in Section 1, Section 3, and Figure 2. We appreciate the valuable suggestion to refine our analysis, and we believe this adjustment enhances the clarity and rigour of our research.

Q2: The pseudocodes of the proposed methods.
A2: In the revised manuscript, we have incorporated the pseudocodes of the proposed methods in Algorithm 1 and Algorithm 2. We sincerely appreciate the valuable suggestion to refine our method, and we believe this adjustment enhances the logical flow of ideas.

Q3: The hyperparameter settings of the proposed methods.
A3: In the revised manuscript, we have included detailed experimental hyperparameters in Table 3. We sincerely appreciate the valuable suggestion to refine our experiment, and we believe this adjustment enhances the clarity and rigor of our research.

Q4: Related work.
A4: Thank you for pointing out this related work (Liu et al., PAS), and we have cited it in the revised manuscript. Similar to our attack strength approach, PAS also adjusts the adversarial budgets. However, it is important to note that there are some key differences:

• Different research problems. PAS investigates the optimization challenges in adversarial training, specifically tackling the increased curvature and gradient scattering caused by the large adversarial budget. Instead, our focus is on the phenomenon of robust overfitting within adversarial training, specifically analyzing the onset of robust overfitting.
• Different perspectives. The main focus/contribution of our work lies in understanding the generation of non-effective features. The attack strength method is designed based on our understanding to specifically eliminate non-effective features. Instead, the design of PAS is motivated directly by the influence of the perturbation budget on loss landscape. Therefore, the two works clearly have different motivations and design principles.
• Different solutions. The two are also different in the strategies for adjusting perturbation budgets. PAS employs a periodic budget scheduling scheme where the perturbation budget dynamically changes along training and is applied to all samples. Instead, ours is much simpler and easy to use: the modified perturbation budget is constant and is applied solely to the small-loss adversarial samples.

Thus, according to these key differences, our method is quite different from PAS and provides new perspectives and solutions for understanding and alleviating robust overfitting.

Thank you once again for your valuable comments, which have helped us improve the clarity and rigour of our research. Hope our elaborations and the revisions made in the new manuscript could address your concerns. Please let us know if there is more to clarify.

Dear Reviewer BeRL, thanks for your time reviewing our paper. We have meticulously prepared a detailed response addressing the concerns you raised. Could you please have a look to see if there are further questions? Your invaluable input is greatly appreciated. Thank you once again, and we hope you have a wonderful day!

Dear Reviewer BeRL, thanks for your time reviewing our paper. We have prepared a detailed response addressing the further concerns you raised. Could you please have a look to see if our response has resolved your concerns? Your invaluable input is greatly appreciated. Once again, thank you, and we wish you a wonderful day!

We have also provided a clear definition of non-effective features in Section 1 and Section 3: “certain non-generalizable robust features emerge in the training set, which we denote as non-effective features” and “Some robust features in the training set may lack generalization. We refer to these features as non-effective features.”

Let $f$ and $h$ be a classifier and feature extractor, and let $f\circ h(x)$ be the model. Features for training set become {$h(x)|x\in D_{train}$}. Then, what is robust features in the training set that may lack generalization? Generalization error is generally defined as the average of errors over test set $D_{test}$ as $E_{x\in D{test}} [R (f\circ h(x))]$, and thus, I cannot understand relationship between generalization and robust features in the training set $D_{train}$.

This is just one example; there are too many similarly unclear or messy arguments. Further replies will not change my score as there are many major issues with the paper.

We thank the reviewer for the insightful comments.

Q1: Lack experimental results to support analysis.
A1: The term "learning state gap" between the training and test sets has been revised to "robustness gap" in our updated manuscript. This change has been made consistently throughout the paper. Additionally, we employ quantitative metrics such as adversarial loss and robust accuracy to measure the model's robustness, providing evidential support for the existence of a robustness gap between the training and test datasets, as illustrated in Figure 1(b). Regarding the generation of non-effective features, our analysis is as follows: during the adversarial training process, the model's adversarial robustness on the training set exceeds that on the test set, as evidenced by the adversarial loss and robust accuracy observed in Figure 1(b). This results in a robustness gap between the training and test data. Considering that adversarial perturbations are generated on-the-fly and adaptively adjusted based on the model's robustness. The robustness gap between the training and test data leads to distinct adversarial perturbation on the robust features in these datasets. The varying degree of adversarial perturbation on the robust features in the training and test data amplify the distribution differences between these two datasets, thereby degrading the generalization of robust features on the training set and facilitating the generation of non-effective features. The increasing non-effective features further exacerbates the robustness gap between training and test data, forming a vicious cycle. These analyses are detailed in our updated manuscript presented in Section 1, Section 3, and Figure 2. We appreciate the valuable suggestion to refine our analysis, and we believe this adjustment enhances the clarity and rigour of our research.

Q2: Reorganize the manuscript.
A2: In the revised manuscript, we have relocated the analysis of existing techniques for mitigating robust overfitting to the appendix to alleviate the reading burden. Additionally, we've incorporated the pseudocodes of the proposed methods in Algorithm 1 and Algorithm 2 to enhance the logical flow of ideas. Thank you for bringing this to our attention.

Q3: Larger datasets like ImageNet.
A3: Due to constraints on computational resources, training adversarially robust models on a large-scale dataset like ImageNet is currently beyond our capacity. However, to the best of my knowledge, the phenomenon of robust overfitting is primarily investigated on standard benchmark datasets such as CIFAR10.

Q4: Highlight the results in Table 2.
A4: In the revised manuscript, we have emphasized the best results in Table 2. Thank you for highlighting this matter.

Q5: Negative effect of OROAT methods.
A5: The proposed methods are specifically designed to validate the underlying mechanisms of robust overfitting, as detailed in Algorithm 1 and Algorithm 2. These methods currently involve adjusting adversarial training at the sample level, which introduce some negative impact on the model's adversarial robustness. We think that by refining adversarial training at the feature level, it should be possible to alleviate these drawbacks. This is an intriguing problem, and we appreciate your attention to this issue.

Q6: Lack empirical comparison.
A6: Indeed, we have conducted experiments on two advanced methods for suppressing robust overfitting, namely AWP and MLCAT. It’s noteworthy that both AWP and MLCAT have already effectively mitigated robust overfitting. The proposed approaches still contribute to a complementary improvement in adversarial robustness. The enhanced adversarial robustness on AWP and MLCAT by our proposed methods further highlights the significance of understanding the underlying mechanisms of robust overfitting.

Thank you once again for your valuable comments, which have helped us improve the clarity and rigour of our research. Hope our elaborations and the revisions made in the new manuscript could address your concerns. Please let us know if there is more to clarify.

Dear Reviewer 6tzX, thanks for your time reviewing our paper. We have meticulously prepared a detailed response addressing the concerns you raised. Could you please have a look to see if there are further questions? Your invaluable input is greatly appreciated. Thank you once again, and we hope you have a wonderful day!

Dear Reviewer 6tzX, thanks for your time reviewing our paper. We have meticulously prepared a detailed response and updated manuscript addressing the concerns you raised. Could you please have a look to see if our response has resolved your concerns? Your invaluable input is greatly appreciated. Once again, thank you, and we wish you a wonderful day!

Q4: Experimental comparison.
A4: Indeed, we have conducted experiments on two advanced methods for suppressing robust overfitting, namely AWP and MLCAT. It’s noteworthy that both AWP and MLCAT have already effectively mitigated robust overfitting. The proposed approaches still contribute to a complementary improvement in adversarial robustness. The enhanced adversarial robustness on AWP and MLCAT by our proposed methods further highlights the significance of understanding the underlying mechanisms of robust overfitting.

Thank you once again for your valuable comments, which have helped us improve the clarity and rigour of our research. Hope our elaborations and the revisions made in the new manuscript could address your concerns. Please let us know if there is more to clarify.

We thank the reviewer for the insightful comments.

Q1: The conclusion is too trivial.
A1: It's worth noting that the main focus/contribution of our work lies in understanding the onset of robust overfitting through a detailed analysis of the generation of non-effective features, rather than drawing the conclusion that non-effective features cause robust overfitting. Regarding the generation of non-effective features, our analysis is as follows: during the adversarial training process, the model's adversarial robustness on the training set exceeds that on the test set, as evidenced by the adversarial loss and robust accuracy observed in Figure 1(b). This results in a robustness gap between the training and test data. Considering that adversarial perturbations are generated on-the-fly and adaptively adjusted based on the model's robustness. The robustness gap between the training and test data leads to distinct adversarial perturbation on the robust features in these datasets. The varying degree of adversarial perturbation on the robust features in the training and test data amplify the distribution differences between these two datasets, thereby degrading the generalization of robust features on the training set and facilitating the generation of non-effective features. The increasing non-effective features further exacerbates the robustness gap between training and test data, forming a vicious cycle. These analyses are detailed in our updated manuscript presented in Section 1, Section 3, and Figure 2. We appreciate the valuable suggestion to refine our analysis, and we believe this adjustment enhances the clarity and rigour of our research.

Q2: Discussion on the specific properties for adversarial training.
A2: In Section 3, we humbly offer the discussions on the generation of non-effective features during the adversarial training process: during adversarial training, adversarial perturbations are generated on-the-fly and adaptively adjusted based on the model’s robustness. In the initial stages of adversarial training, due to the similarity in the model's robustness between the training and test datasets, the generated adversarial perturbations on the robust features are relatively close in these datasets. Thus, most robust features in the training set can still exhibit good generalization, thereby enhancing the model's adversarial robustness on the test set. However, as training progresses, the model's robustness in the training dataset increases significantly faster than its robustness on the test dataset, as evidenced in Figure 1(b) by the adversarial loss and robustness observed on both datasets. This leads to a widening robustness gap between the training and test datasets. Consequently, the generated adversarial perturbation on the robust features becomes progressively more distinct in these datasets, which degrades the generalization of robust features on the training set and facilitates the generation of non-effective features, causing the model to learn an increasing number of non-effective features. As non-effective features proliferate, the robustness gap between the training and test sets continues to grow, forming a vicious cycle, as illustrated in Figure 2. Once the model's optimization is governed by these non-effective features, the model's adversarial robustness on the test dataset will continue to decline. This, in turn, gives rise to the phenomenon of robust overfitting.

Q3: The methods are not interesting.
A3: We regret that the proposed methods did not capture your interest. In fact, our proposed methods are specifically crafted to validate the underlying mechanisms of robust overfitting, as outlined in Algorithm 1 and Algorithm 2. It is worth noting that, while data augmentation is almost the most natural way to suppress overfitting, applying the same data augmentation techniques in adversarial training, following the standard practice, proves ineffective in mitigating robust overfitting. Similarly, although attack strength adjustment is common for adversarial training, employing the same attack strength adjustment in accordance with conventional manner does not alleviate robust overfitting. In summary, while our methods may seem relatively simple and common, they are purposefully designed to validate our analysis of robust overfitting. Furthermore, they consistently enhance adversarial robustness across a wide range of experimental settings, demonstrating their efficacy.

Dear Reviewer sV3q, thanks for your time reviewing our paper. We have meticulously prepared a detailed response addressing the concerns you raised. Could you please have a look to see if there are further questions? Your invaluable input is greatly appreciated. Thank you once again, and we hope you have a wonderful day!

Dear Reviewer sV3q, thanks for your time reviewing our paper. We have prepared a detailed response addressing the further concerns you raised. Could you please have a look to see if our response has resolved your concerns? Your invaluable input is greatly appreciated. Once again, thank you, and we wish you a wonderful day!

We thank the reviewer for the insightful comments.

Q1: Factor ablation experiments over a few trials.
A1: The results of factor ablation experiments in Figure 1 show the outcomes of a single run. However, it is worth noting that we have actually presented the results of these factor ablation experiments across different datasets, network architectures, and adversarial training variants, as illustrated in Figure 3. This broader presentation suggests that the observed phenomenon is a general finding in adversarial training.

Q2: No algorithm for the proposed methods.
A2: In the revised manuscript, we have incorporated the pseudocodes of the proposed methods in Algorithm 1 and Algorithm 2. We sincerely appreciate the valuable suggestion to refine our method, and we believe this adjustment enhances the clarity of our research.

Q3: No analysis to support the claims about robust overfitting.
A3: The term "learning state gap" between the training and test sets has been revised to "robustness gap" in our updated manuscript. This change has been made consistently throughout the paper. Additionally, we employ quantitative metrics such as adversarial loss and robust accuracy to measure the model's robustness, providing evidential support for the existence of a robustness gap between the training and test datasets, as illustrated in Figure 1(b). Regarding the process of how the robustness gap induces robust overfitting, our analysis is as follows: during the adversarial training process, the model's adversarial robustness on the training set exceeds that on the test set, as evidenced by the adversarial loss and robust accuracy observed in Figure 1(b). This results in a robustness gap between the training and test data. Considering that adversarial perturbations are generated on-the-fly and adaptively adjusted based on the model's robustness. The robustness gap between the training and test data leads to distinct adversarial perturbation on the robust features in these datasets. The varying degree of adversarial perturbation on the robust features in the training and test data amplify the distribution differences between these two datasets, thereby degrading the generalization of robust features on the training set and facilitating the generation of non-effective features. The increasing non-effective features further exacerbates the robustness gap between training and test data, forming a vicious cycle. These analyses are detailed in our updated manuscript presented in Section 1, Section 3, and Figure 2. We appreciate the valuable suggestion to refine our analysis, and we believe this adjustment enhances the clarity and rigour of our research.

Q4: The definition of "effective features".
A4: The term "effective features" has been deprecated in our revised manuscript. We now explicitly denote them as robust features in the training dataset with generalization. For instance, "In the initial stages of adversarial training, due to the similarity in the model's robustness between the training and test datasets, the generated adversarial perturbation on the robust features is relatively close in these datasets. Thus, most robust features in the training set can still exhibit good generalization, thereby enhancing the model's adversarial robustness on the test set." We appreciate the feedback and have made the necessary adjustments to improve clarity.

Q5: The error in Eqn(4).
A5: We have rectified the error in Eqn (4) in the revised manuscript. Thank you for bringing it to our attention.

Thank you once again for your valuable comments, which have helped us improve the clarity and rigour of our research. Hope our elaborations and the revisions made in the new manuscript could address your concerns. Please let us know if there is more to clarify.

Dear Reviewer H4Yh, thanks for your time reviewing our paper. We have meticulously prepared a detailed response addressing the concerns you raised. Could you please have a look to see if there are further questions? Your invaluable input is greatly appreciated. Thank you once again, and we hope you have a wonderful day!

Dear Reviewer H4Yh, thanks for your time reviewing our paper. We have meticulously prepared a detailed response and updated manuscript addressing the concerns you raised. Could you please have a look to see if our response has resolved your concerns? Your invaluable input is greatly appreciated. Once again, thank you, and we wish you a wonderful day!

We thank the reviewer for the positive and constructive comments regarding our paper.

Q1: Notion of learning state gap.
A1: The term "learning state gap" between the training and test sets has been revised to "robustness gap" in our updated manuscript. This change has been made consistently throughout the paper. Additionally, we employ quantitative metrics such as adversarial loss and robust accuracy to measure the model's robustness, providing evidential support for the existence of a robustness gap between the training and test datasets, as illustrated in Figure 1(b). Regarding the process of how the robustness gap induces robust overfitting, our analysis is as follows: during the adversarial training process, the model's adversarial robustness on the training set exceeds that on the test set, as evidenced by the adversarial loss and robust accuracy observed in Figure 1(b). This results in a robustness gap between the training and test data. Considering that adversarial perturbations are generated on-the-fly and adaptively adjusted based on the model's robustness. The robustness gap between the training and test data leads to distinct adversarial perturbation on the robust features in these datasets. The varying degree of adversarial perturbation on the robust features in the training and test data amplify the distribution differences between these two datasets, thereby degrading the generalization of robust features on the training set and facilitating the generation of non-effective features. The increasing non-effective features further exacerbates the robustness gap between training and test data, forming a vicious cycle. These analyses are detailed in our updated manuscript presented in Section 1, Section 3, and Figure 2. We appreciate the valuable suggestion to refine our analysis, and we believe this adjustment enhances the clarity and rigour of our research.

Q2: Different data augmentation techniques.
A2: Indeed, we conducted experiments with four different data augmentation techniques, as shown in Figure 1(d) and Figure 6, all of which demonstrated similar effects on robust overfitting. This consistency arises from the fact that data augmentation techniques are applied with a targeted transformation objective to modify the proportion of small-loss adversarial data, as outlined in Algorithm 2. Consequently, they inherently achieve the effect of robustness alignment, and the specific data augmentation techniques employed do not significantly impact their effectiveness.

Q3: Large-scale dataset like ImageNet.
A3: Due to constraints on computational resources, training adversarially robust models on a large-scale dataset like ImageNet is currently beyond our capacity. As far as I am aware, the phenomenon of robust overfitting is predominantly investigated on standard benchmark datasets such as CIFAR10. I regret to admit my uncertainty regarding potential differences in the robust overfitting phenomenon between the CIFAR10 and ImageNet datasets.

Thank you once again for your valuable comments, which have helped us improve the clarity and rigour of our research. Hope our elaborations and the revisions made in the new manuscript could address your concerns. Please let us know if there is more to clarify.