Identifying and Understanding Cross-Class Features in Adversarial Training

Dear Reviewer v6k6,

Thank you for your valuable feedback. We address your concerns below.

---

Q1: A more solid explanation for Figure 8 is needed

A1: Thank you for your thoughtful comment. First, we would like to clarify that Figure 8 aims to show that our theory for adversarial training (AT) is compatible with standard training (ST), where the overfitting issue is far less than robust overfitting in AT. This is supported by the fact that both the accuracy and CAS of ST do not change as significantly as AT in the latter training stage. As for the overfitting in ST stated in [1], it may have other mechanisms like sample memorization [A], but we kindly note that such understandings are not within the scope of our study, since our theory focuses on particular properties of AT. We will add this discussion in our revision.

[A] The Pitfalls of Memorization: When Memorization Hurts Generalization. ICLR 2025

---

Q2: The potential use of cross-class features

A2: Thanks for the insightful comments. Due to space limitations, please kindly refer to our response to Q1 by Reviewer HgFV for a discussion on the future directions of cross-class features.

---

Q3: Regarding catastrophic overfitting

A3: Thank you for the insightful comment. Following your suggestion, we implemented a 200-epoch fast adversarial training with 1 step PGD attack, $\ell_\infty$-norm $\epsilon=8/255$ on PreActResNet-18 (same as the main observation experiment in our paper). The CAS for different stages is presented as follows:

The results also validate that the forgetting of cross-class features (measured by CAS) can be detected in the catastrophic overfitting of fast adversarial training, which aligns with the main discovery of our paper. We will include these results and related code, figures in our revision. Thanks again for raising this point!

---

Q4: Class names displayed in the figures

A4: Thanks for the kind suggestion. We will add these class labels to the saliency maps in our revision.

---

Q5: Please ensure consistency in notation.

A5: Thanks for the careful reading. We will unify them as $A_i$ in our revision.

---

We truly appreciate your valuable and constructive feedback. If you have any further questions or concerns, please let us know.

Dear Reviewer RcTN,

Thank you for your valuable feedback. We address your concerns below.

---

Q1: A concern I have is the paper’s discussion on its significance for future research. While previous work may not have fully understood how AT utilizes class features, we still have an intuitive understanding of the underlying logic. Even with the clearer evidence provided, the challenge remains in how to preserve reliance on cross-class features while continuing to improve the robustness of the model.

A1: Thank you for the thoughtful comment. We list some potential directions for further applications of our theory as follows:

• Data (re)sampling. Since more generated data are prone to be helpful for advancing adversarial robustness [1], it requires significantly more data and computational costs. From the cross-class feature perspective, adaptively sampling generated data with considerations of class-wise relationship may improve the efficiency of large-scale AT and decrease the forgetting of cross-class features.
• AT configurations. Customizing AT configurations like perturbation margins or neighborhoods is useful for improving robustness [2]. Since cross-class features are more sensitive against robust loss, customizing AT configurations for different samples or classes based on class-wise relationships may mitigate this sensitivity and further improve robustness, as shown in Theorem 1.
• Robust module design: The architecture of a model and the mechanisms of activation play a crucial role in adversarial robustness [3]. Therefore, designing modules that either implicitly or explicitly emphasize cross-class features may enhance robustness. For example, calibrating channel activation can improve robustness [4], thus creating activation mechanisms that preserve more cross-class features can further contribute to this improvement.

In addition to these AT algorithms, we would also like to highlight the theoretical modeling potential of our work. Similar to the robust/non-robust feature decomposition [5], which has been applied in many subsequent theoretical works, e.g. [6,7,8], our cross-class feature model has the potential for more in-depth modeling of adversarial robustness, contributing new tools in its theoretical analysis.

[1] Better Diffusion Models Further Improve Adversarial Training. ICML 2023

[2] CAT: Customized Adversarial Training for Improved Robustness. IJCAI 2022

[3] Robust Principles: Architectural Design Principles for Adversarially Robust CNNs. BMVC 2023

[4] Improving Adversarial Robustness via Channel-wise Activation Suppressing. ICLR 2021

[5] Adversarial examples are not bugs, they are features. NeurIPS 2019

[6] On the Tradeoff Between Robustness and Fairness. NeurIPS 2022

[7] Understanding the Impact of Adversarial Robustness on Accuracy Disparity. ICML 2023

[8] Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data. ICLR 2025

---

Q2: I suggest that the authors provide more detailed explanations of the concepts introduced in their paper. For instance, the terms "robust accuracy" and "robust loss" are used without further clarification. I assume these terms refer specifically to the accuracy and loss measured on adversarial examples, but it would be helpful for the authors to explicitly define them for clarity.

A2: Thanks for your careful reading. Your assumption is correct, we will clarify these definitions in our revision as follows:

• Robust accuracy is the accuracy of the model on adversarial examples.
• Robust loss is the average cross-entropy loss of the model on adversarial examples.

---

We truly appreciate your valuable and constructive feedback. If you have any further questions or concerns, please let us know.

Dear Reviewer oCW6,

Thank you for your valuable feedback. We address your concerns below.

---

Q1: Implementation details

A1: Thank you for your careful reading. We are committed to publishing our code upon publication. For implementation details regarding model training, we utilize the default hyperparameters according to a well-known adversarial training repository [1] listed below:

We will make these details clear in our revision. Thanks again for the reminder!

[1] Bag of Tricks for Adversarial Training, ICLR 2021, https://github.com/P2333/Bag-of-Tricks-for-AT.

---

Q2: Saliency map visualizations.

A2: Thank you for your thoughtful comment. To show these examples are not cherry-picked, we will include a full page of visualization examples (ordered by original sample ID) in the appendix of our revision, where many saliency maps of these examples still exhibit such properties. However, we acknowledge that not all samples enjoy such clearly interpretable features (e.g., wheels shared by automobiles and trucks), since features learned by neural networks are subtle and do not always align with human intuition, including cross-class features. Thus, we will tone down related claims made in that paragraph and highlight that the saliency maps are only presented to help understand the concept of cross-class features.

---

Q3: The theoretical data model in Section 4 is fixed to 6 dimensions. What was the reason for choosing a 6-dimensional data representation? Would any complications arise when trying to extend the theoretical results to higher dimensions?

A3: Thanks for the thoughtful comment. First, we acknowledge that using a 6-dimensional data representation is the smallest dimension for illustrating our insights into cross-class features, since binary classification is not capable of handling cross-class features as they do not influence the binary classification result. Therefore, we explored a ternary classification problem with minimum dimensions in our framework.

Following your suggestion, we attempted to extend our theory to more feature dimensions for ternary classification . For each original feature $x_{E,j}$ or $x_{C,j}$, we can extend them to $x_{E,j}^k$ or $x_{C,j}^k$, where $k=1,2,\cdots,K$, thus resulting in $6K$ feature dimensions. Accordingly, we also have corresponding parameters $w_1^k$ and $w_2^k$ for $k=1,2,\cdots, K$. Based on this extended model, we can derive similar results in Theorems 1-3, where the bounds are set for $w_1^k$ and $w_2^k$. The proof sketch includes deriving the optimal perturbation $\epsilon$ with Lemma 2, calculating the optimizing objectives like equation (26), and finally derive the solution of $w_1^k$ and $w_2^k$ for minimizing the objectives like equation (27) and (29). Due to space limitations, we are unable to present all proof details here, but we will include this extension in the appendix of our revision. Thank you again for the valuable suggestion!

---

Q4: The term "feature" feels overloaded in different sections of this paper.

A4: Thank you for your careful reading. We clarify the definition of “features” as the activation in a particular layer (feature extractor), as stated in Section 3.1. Thus, the feature is model-dependent under this definition.

Regarding saliency maps and Section 4, we apologize for the potential confusion and will replace the term “features” as follows. For saliency maps, we will use the term “class-specific discriminative regions” that was used in the original paper of [2] GradCAM for the highlighted regions. For $x_{E,j}$ and $x_{C,j}$ in Section 4, we will define them as “attribution” instead of features to distinguish them. Thanks again for your kind reminder, we will revise these terms in our revision.

[2] Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. IJCV 2019

---

We truly appreciate your valuable and constructive feedback. If you have any further questions or concerns, please let us know.

Dear Reviewer HgFV,

Thank you for your valuable feedback. We address your concerns below.

---

Q1: It would be better if authors could discuss, based on the findings in this work, any possible ways to develop advanced adversarial training methods.

A1: Thank you for the thoughtful comment. We list some potential directions for further applications of our theory as follows:

• Data (re)sampling. Since more generated data is prone to be helpful for advancing adversarial robustness [1], it requires significantly more data and computational costs. From the cross-class feature perspective, adaptively sampling generated data with considerations of class-wise relationship may improve the efficiency of large-scale AT and decrease the forgetting of cross-class features.
• AT configurations. Customizing AT configurations like perturbation margins or neighborhoods is useful for improving robustness [2]. Since cross-class features are more sensitive against robust loss, customizing AT configurations for different samples or classes based on class-wise relationships may mitigate this sensitivity and further improve robustness, as shown in Theorem 1.
• Robust module design: The architecture of a model and the mechanisms of activation play a crucial role in adversarial robustness [3]. Therefore, designing modules that either implicitly or explicitly emphasize cross-class features may enhance robustness. For example, calibrating channel activation can improve robustness [4], thus creating activation mechanisms that preserve more cross-class features can further contribute to this improvement.

In addition to these AT algorithms, we would also like to highlight the theoretical modeling potential of our work. Similar to the robust/non-robust feature decomposition [5], which has been applied in many subsequent theoretical works, e.g. [6,7,8], our cross-class feature model has the potential for more in-depth modeling of adversarial robustness, contributing new tools in its theoretical analysis.

[1] Better Diffusion Models Further Improve Adversarial Training. ICML 2023

[2] CAT: Customized Adversarial Training for Improved Robustness. IJCAI 2022

[3] Robust Principles: Architectural Design Principles for Adversarially Robust CNNs. BMVC 2023

[4] Improving Adversarial Robustness via Channel-wise Activation Suppressing. ICLR 2021

[5] Adversarial examples are not bugs, they are features. NeurIPS 2019

[6] On the Tradeoff Between Robustness and Fairness. NeurIPS 2022

[7] Understanding the Impact of Adversarial Robustness on Accuracy Disparity. ICML 2023

[8] Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data. ICLR 2025

---

Q2: As authors' claimed, after some checkpoint, the trained model reduce its reliance on cross-class features, which result in the reduced robustness on test data; so could we understand this phenomenon in another way, i.e. some class-specific features are naturally contradicted with cross-class features, as cross-class features are helpful for model robustness, the robustness of model will be reduced if model starts learning those contradicted class-specific features after some checkpoint.

A2: Thank you for the alternative interpretation of our theory, which aligns well with our understanding. There may indeed be some contradiction between cross-class and class-specific features, caused by various factors such as model capacity or feature overlap. As a result, learning more class-specific features for lower training robust loss can lead to a decrease in cross-class features, as discussed in Section 3.2. Thank you once again for your insightful comment; we will incorporate this discussion into our revision.

---

We truly appreciate your valuable and constructive feedback. If you have any further questions or concerns, please let us know.