ASTrA: Adversarial Self-supervised Training with Adaptive-Attacks

We thank reviewer w4mR for appreciation and insightful reviews.

[W1] Readability issues

Thank you for pointing out on reference in Table 1 and other concerns including clean loss. We have corrected the same in manuscript also improved the readability by revisiting the content and providing added explanation of method with easy-to-understand visualization and step-wise description in appendix.

[W2, Q1] Interaction between the target and strategy networks and sampling from the conditional distribution

Interaction between Strategy and Target Networks in ASTrA

Figure 8 (Please refer Section A.1 in appendix of updated manuscript) illustrates the ASTrA framework involving an adaptive attack strategy network interacting with a target network in an adversarial training setup. The framework focuses on dynamically crafting adversarial examples and aligning the clean and adversarial representations for robust learning, explained in the following steps.

Step 1: The strategy network generates probability distributions for each perturbation parameter from which iteration ($i$), epsilon ($e$), and step size ($s$) are sampled. The perturbation (attack) parameters are sampled according to a conditional probability distribution, given by $p(a|x,\theta)$.

Step 2: ThThese perturbed parameters ($i$, $e$, $s$) are used to craft adversarial examples from the input images for the target network. The target network feed-forwards both clean and adversarially perturbed images and computes three loss terms:

• (a) Contrastive loss on clean image views ($L_{\text{clean}}$)
• (b) Contrastive loss on perturbed views ( $L_{\text{adv}}$), and
• (c) Contrastive loss on clean to perturbed views ($L_{\text{mixed}}$)

These loss terms are back-propagated to the target network.

Step 3: The clean loss (a) and adversarial loss (b) terms are used to compute the reward for the strategy network. This computed reward is fed to the REINFORCE [1, Williams, 1992] algorithm to compute gradients required to update the strategy network. The sampling process $( a \sim p(a|x; \theta) )$ is not differentiable with respect to $( \theta )$, making traditional gradient-based optimization methods inapplicable. The REINFORCE algorithm is specifically designed to handle such situations by using the log-derivative trick, enabling gradient-based optimization even when sampling is involved. The objective of the strategy network is to maximize the expected reward $J(\theta)$, defined as:

$

  J(\theta) = \mathbb{E}{x \sim D} \left[ \mathbb{E}{a \sim p(a|x; \theta)} \left[ R_{\text{strategy}}(x, a; \theta) \right] \right].
  

$

- To compute the gradient of $J(\theta)$ with respect to $\theta$, we use the property of probability distributions:

$

\nabla_\theta p(a|x; \theta) = p(a|x; \theta) \nabla_\theta \log p(a|x; \theta).

$

The gradient of $J(\theta)$ is expanded as:

$

\nabla_\theta J(\theta) = \mathbb{E}{x \sim D} \left[ \nabla\theta \mathbb{E}{a \sim p(a|x; \theta)} \left[ R{\text{strategy}}(x, a; \theta) \right] \right].

$

Using the log-derivative trick, the inner gradient is expressed as:

$

\nabla_\theta \mathbb{E}{a \sim p(a|x; \theta)} \left[ R{\text{strategy}} \right] = \mathbb{E}{a \sim p(a|x; \theta)} \left[ R{\text{strategy}} \nabla_\theta \log p(a|x; \theta) \right].

$

Substituting back, the gradient of $J(\theta)$ becomes:

$

\nabla_\theta J(\theta) = \mathbb{E}{x \sim D} \left[ \mathbb{E}{a \sim p(a|x; \theta)} \left[ R_{\text{strategy}}(x, a; \theta) \nabla_\theta \log p(a|x; \theta) \right] \right].

$

The reward $R_{\text{strategy}}$ scales the gradient update, encouraging the strategy network to favor attack parameters $a$ that yield higher rewards. This ensures that the strategy network learns attack strategies that balance adversarial robustness and generalization.

[1] Williams, Ronald J. "Simple statistical gradient-following algorithms for connectionist reinforcement learning." Machine learning 8 (1992): 229-256.

We sincerely thank you for your detailed review and insightful feedback on our paper.

Summary of Responses:

• We have addressed all the readability concerns, clarified the interaction between the target and strategy networks, and provided additional computational complexity analysis. Specifically:
  • Corrected references and equations to improve the clarity of the manuscript.
  • Added detailed explanations and step-by-step interaction between the target and strategy networks, including how attack parameters are sampled from the conditional distribution and the role of reward optimization using REINFORCE.
  • Provided computational overhead analysis to highlight ASTrA's efficiency compared to existing methods.
  • Included comparisons with state-of-the-art methods like DynACL++ and DynACL-AIR++ to demonstrate ASTrA’s strengths.

Satisfaction with Responses:

• We believe our responses and added experiments effectively address the concerns. The paper has been revised for clarity, and all questions have been answered comprehensively with supporting empirical evidence.

Request for Score Improvement:

• Based on our thorough responses and improvements to the manuscript, we kindly request you to consider increasing the score for our submission. We are happy to address any further questions or suggestions.

Thank you for your thoughtful and constructive feedback.

We would like to thank the reviewer for their thoughtful feedback. We are responding to ensure there is no misunderstanding regarding what ASTrA and ASTrA++ represent and the SLF/AFF evaluation results.

1. ASTrA++ Pretraining:
   ASTrA++ employs longer pretraining (2000 epochs), whereas DynACL++ and DynACL-AIR++ utilize a post-processing phase (clustering followed by Pseudo Adversarial Training). These approaches are fundamentally different, and ASTrA++ results do not involve post-processing.

2. Evaluation Results:
   ASTrA

   • ASTrA achieved state-of-the-art performance across all datasets (CIFAR-10,CIFAR-100,STL-10) in both SLF and AFF evaluations when compared to DynACL and DynACL-AIR, highlighting the robustness of our approach.

   ASTrA++

   • ASTrA++ achieves state-of-the-art performance on CIFAR-100 and STL-10 datasets in both SLF and AFF evaluations, highlighting its robustness. Unlike DynACL++ and DynACL-AIR++, our approach ASTrA++ does not apply any post-processing methods and demonstrates competitive results solely through extended pretraining. We note that ASTrA++ could also benefit from post-processing; however, such modifications are not reported to comply with ICLR guidelines prohibiting post-submission changes.

3. Robustness:
   The results demonstrate that our method achieves strong representation robustness solely through pretraining, without relying on post-processing, underscoring the effectiveness of the longer pretraining approach.

We hope this clarification resolves any ambiguity regarding the robustness achieved by our method. Thank you again for your constructive feedback and engagement. We are looking forward for score upgrades and happy to address any further queries or suggestions.

We thank reviewer veWg for appreciation and insightful reviews.

[W1] Training complexity of ASTrA

Effect of strategy network on training stability

The incorporation of the strategy network makes the training process adaptive and stable by dynamically adjusting attack parameters at the instance level, leveraging the learning dynamics of the target network at each step through observations of standard and adversarial loss terms. This stability is evident based on the smoothness of the loss landscape visualization in Figure 12 (section A.5 in appendix). Additionally, attack parameter bin configurations identified empirically for CIFAR-10 are successfully reused for STL-10 and ImageNet100, demonstrating that the model is not sensitive to initial bin configurations and consistently adapts toward convergence.

Computation analysis of ASTrA

The Table below highlights the computation analysis of onboarding different strategy networks within the ASTrA framework. The results show that adding a strategy network introduces a slight increase in computation time. For instance, the compute time increases from 20.5 hours for the ACL baseline (which lacks a strategy network) to 23.4 hours for ResNet18, which is the largest architecture in this analysis. This represents an additional compute overhead of less than 3 hours. For smaller networks such as MobileNetV1 or CustomCNN, the increase in compute time is even smaller, around 1 to 1.5 hours. These results indicate that the computational overhead introduced by the strategy network remains minimal and manageable in all cases.

Table: Ablation (SLF evaluation) on Strategy network choices. Compute-robustness trade-off. Training conducted on single H100 GPU. ASTrA tends to be network-agnostic for the strategy network, and even with smaller architectures, it outperforms ACL. Choices of networks like ResNet10, EfficientNet-B0, and DenseNet-121 achieve SoTA.

Referring Table above, ASTrA proves to be network-agnostic, achieving consistent performance improvements across both parametric and non-parametric strategy network choices. Regardless of the architecture, ASTrA outperforms the ACL baseline, showcasing its versatility and robustness. Smaller, lightweight architectures such as MobileNetV1 and CustomCNN still achieve competitive performance, while larger architectures such as ResNet10, EfficientNet-B0, and DenseNet-121 improve state-of-the-art results.

The results also reveal a positive trend where increasing the complexity of the strategy network leads to incremental gains in adversarial and robust accuracy metrics. This demonstrates ASTrA’s ability to leverage the capacity of various strategy networks effectively, reinforcing its robustness and generalization capabilities while keeping computational overhead minimal.

We sincerely thank you for your detailed review and insightful feedback on our paper.

Summary of Responses:

• We have addressed all questions and provided empirical evidence for ASTrA's effectiveness. Specifically:
  • Detailed analysis of training complexity and stability, showing that ASTrA introduces minimal computational overhead and remains stable across training setups.
  • Comprehensive experiments on the discretization of attack parameters, demonstrating the trade-offs between efficiency and performance.
  • Clarifications on how ASTrA could be adapted for supervised settings and how exploration transitions into exploitation in the training process.
  • Improved presentation of Figure 8 by replacing it with a table and contextualizing ASTrA's relationship to adversarial curriculum learning.

Satisfaction with Responses:

• We believe our responses effectively address the concerns raised. We have added clarifications and empirical insights to enhance the paper’s contributions and its alignment with your suggestions.

Request for Score Improvement:

• Based on our thorough responses and updates, we kindly request you to consider increasing the score for our submission. We are available to address any further queries or suggestions.

Thank you for your thoughtful and constructive feedback.

We thank reviewer veWg for their efforts for going through our response and increasing score.

We thank reviewer iFgE for appreciation and insightful reviews.

[W1] - empirical experiments on larger scale

Despite CIFAR10, CIFAR100, and STL-10 being supervised datasets, ASTrA utilizes them without labels during the pretraining stage, as it is a self-supervised adversarial training method. The primary objective of using these datasets is to enable a fair comparison with existing self-supervised adversarial training methods, as most prior works use these datasets as baselines.

In Section 4.3, ASTrA's scalability is evaluated by applying it to the larger ImageNet-100 dataset, which contains 120k images with a resolution of 224x224, compared to CIFAR10's 32x32 resolution and STL-10's 96x96 resolution. ASTrA continues to demonstrate consistent robustness on the ImageNet-100 dataset, further validating its scalability and effectiveness on larger, more complex datasets.

[Q1] The last term of reward (eq (4)) does not contain the optimized parameter $\theta$, why is it necessary to add it to the reward function?

Role of $L_{\text{clean}}$ and $L_{\text{adv}}$

The reward function $R_{\text{strategy}}$ incorporates both $L_{\text{adv}}$ and $L_{\text{clean}}$:

Here:

• $L_{\text{adv}}$ depends on $\theta$, as adversarial examples $x_{\text{adv}}$ are crafted using attack parameters $a$, which are influenced by $\theta$.
• $L_{\text{clean}}$ does not depend on $\theta$, as it is computed using clean data and the fixed target network $w_{\text{fixed}}$.

Substituting $R_{\text{strategy}}$ into the gradient that is computed using REINFORCE algorithm using the log-derevative trick:

Effect of $L_{\text{clean}}$

While $L_{\text{clean}}$ does not directly depend on $\theta$, we can see that it indirectly affects the updates to $\theta$ by scaling the reward $R_{\text{strategy}}$:

• Large $L_{\text{clean}}$ reduces $R_{\text{strategy}}$, discouraging attack strategies that degrade clean performance.
• Small $L_{\text{clean}}$ increases $R_{\text{strategy}}$, reinforcing attack strategies that preserve generalization.

This ensures that the strategy network learns attack parameters $a$ that balance robustness (via $\alpha L_{\text{adv}}$) and generalization (via $\gamma L_{\text{clean}}$).

Our method is detailed further in section A.1 figure 8 in appendix in manuscript with details including ASTrA’s steps, strategy network’s role, clean loss in rewards, and training stability.

[Q2] non-parametric modeling

In ASTrA, the ability to create adaptive adversarial attacks relies heavily on the REINFORCE algorithm, which requires a differentiable, parameterized policy to compute gradients. This setup allows the Strategy Network to learn and update its attack strategies based on reward signals from the Target Network's performance. Non-parametric models, however, lack fixed parameters that can be adjusted through gradient computations. This makes them incompatible with the gradient-based optimization that REINFORCE depends on. Without the capacity to update parameters via gradients, non-parametric methods can't adapt attack strategies in response to the changing state of the Target Network. This incompatibility disrupts the core mechanism of ASTrA, where dynamic adjustment of attack strategies is crucial for effective adversarial training.

While Gaussian Processes [1] offer a non-parametric approach that can, in theory, provide adaptable function approximations, they are impractical for ASTrA's context due to their computational complexity. Gaussian Processes have a computational cost that scales cubically with the number of data points ($\mathcal{O}(n^3)$), making them unsuitable for large datasets common in adversarial training of image models. Additionally, integrating Gaussian Processes into the REINFORCE framework is challenging because computing gradients in high-dimensional spaces (such as those of image data) is computationally intensive and inefficient. Therefore, despite their theoretical capability to enable adaptation, Gaussian Processes are not feasible for ASTrA given the dataset sizes and the need for scalable modeling.

[1] Williams, C. K., & Rasmussen, C. E. (2006). Gaussian processes for machine learning (Vol. 2, No. 3, p. 4). Cambridge, MA: MIT press.

We have added additional computational analysis for different choices of strategy networks is presented in section A.4.1 table 7 in appendix of manuscript.

We sincerely thank you for your detailed review and insightful feedback on our paper.

Summary of Responses:

• We have addressed all questions raised and provided empirical evidence supporting our method. Specifically:
  • Empirical experiments on larger-scale datasets have been included to demonstrate ASTrA's scalability and effectiveness. We evaluated ASTrA on the ImageNet-100 dataset, which is significantly larger and more complex, further validating its performance.
  • The necessity of including the term $L_{\text{clean}}$ in the reward function was clarified, showing how it balances robustness and generalization without direct dependence on $\theta$, ensuring effective adversarial training.
  • The potential for non-parametric modeling was discussed, with a detailed explanation of why it is not compatible with ASTrA's framework and objectives.

Satisfaction with Responses:

• We believe our responses and additional empirical analysis effectively address the concerns. We provided extensive clarifications, including theoretical justifications and new references, and demonstrated ASTrA's robustness and adaptability across varying conditions.

Request for Score Improvement:

• Based on our thorough responses and additional experiments, we kindly request you to consider increasing the score for our submission. We are available to address any further questions or concerns you may have.

Thank you for your time and valuable feedback.

Thanks for the response of the authors. I do not have any major concern about the paper. I think large-scale experiments on unsupervised data on datasets like tiny image could demonstrate the power of ASTrA. Without such experiment, I think the contribution of paper is above the acceptance threshold but lacks of major contribution that further increases the score.