Adapting to Evolving Adversaries with Regularized Continual Robust Training

Thank you for your insightful review.

Discussion of RAMP

Thank you for pointing us to this interesting relevant work. This work looks at achieving robustness against multiple Lp norms and proposes a logit pairing loss which aims to minimize the KL divergence between the logits of predicting on 2 different Lp attacks. Additionally, they use gradient projection to integrate model updates between natural training and adversarial training for better clean accuracy-robustness tradeoff. In comparison, our work looks at robustness against sequences of attacks including non-Lp attacks. Our regularization term uses $\ell_2$ distance between clean and adversarial logits. We will add this discussion into Appendix A and add a citation to the RAMP paper into Section 5.

Significance

We believe that it is reasonable to model the defender’s knowledge of potential threats in a sequential manner. Firstly, it takes time for researchers and attackers alike to develop new attacks. For example, the UAR benchmark [1] was initially released in 2019 and was then expanded with new non-Lp attacks in 2023. When new attacks are discovered, the defender would want to quickly adapt their model for robustness. Additionally, in the case that multiple attacks are discovered simultaneously, our regularized CRT can be used with multiple attacks at a single timestep. For example, in initial training, we can use existing methods for multiattack training and add ALR with respect to all attacks, and in finetuning, we can use a finetuning strategy such as FT Croce + ALR which is able to take into account multiple attacks.

[1] Kaufmann et al. (2019). Testing robustness against unforeseen adversaries. arXiv preprint

Real world attack setting

We discuss challenges of extending to real-time attacks in the “Extension to scenarios where defender has limited knowledge about the attack type” section of our response to Reviewer 9HKR.

Does the order in which different attacks occur impact the final results of the proposed methods?

We provide results for another ordering of the 4 attacks (Linf->StAdv->Recolor->L2) in Appendix Table 5. Overall, we observe the same trends with ALR helping in reducing forgetting of robustness on previous attacks and improving robustness on held out attacks. The final model obtained after all rounds of finetuning for this sequence achieves higher union all accuracy (3.32% higher than the sequence in the main paper) so order may have some impact on the final model performance. This is also shown in the finetuning ablations in Fig 3 as the matrix is not symmetric.

Which term is regularized in Theorem 3.1

For initial training with ALR, only the adversarial loss term corresponding to the initial known attack is regularized. In finetuning, the term regularized depends on the finetuning strategy. We use regularization with the attack that is selected to be computed on the batch by the finetuning strategy. Specifically, if we have $\mathcal{L}_1$ be the loss on the previous attack and $\mathcal{L}_2$ be the loss on the new attack used in finetuning, then for FT Single, only the second term is regularized since we only use the new attack in finetuning. For FT Croce, both terms are regularized across training since both attacks have a chance of being chosen during training (although we are only computing regularization with respect to a single attack per batch).

Novelty and future directions

We acknowledge that there are similarities between our ALR regularizer and TRADES regularizer as they both maximize a distance in the logit space, but we highlight that ALR is theoretically motivated from the standpoint of improving generalization to new/unforeseen attacks and reducing forgetting of previous attacks while TRADES is motivated from the standpoint of balancing clean accuracy-robustness tradeoff. We also highlight that the study of obtaining robustness against sequences of different attack threat models is novel and we contribute extensive experiments across a variety of attack types and investigate the performance of random-noise based regularization as well.

For an experimental comparison to TRADES, please refer to the “comparisons with TRADES” portion of response to reviewer 1uU6. We observe that ALR works better for our task as it does not trade off robustness on the initial attack.

We discuss future directions for enhancing this line of work in Appendix B. These include ways of detecting attacks, further improving finetuning efficiency, studying the impact of model capacity, and theoretical analysis comparing loss under different attacks for the initial model after training and model obtained after finetuning.

L1 attack

Please see our response to Reviewer JuEE portion on “L0 Attack” for explanation as to why we used the set of evaluation attacks used in the paper. If the reviewer thinks it is necessary, we can add a few experiments with L1 attack for the final version of the paper.

General clarifications

Goals in CAR

We optimize three objectives (Def 2.1): (1) robustness to known attacks, (2) robustness to unforeseen attacks, and (3) update efficiency. (Known) metrics correspond to (1), (all) metrics to (2), and training time to (3). Thus, (known), (all), and time metrics are all key for comparing techniques. MAX and FT-MAX may achieve better accuracy but are less efficient.

Finetuning Ablations

We direct the reviewer to Fig 3 and discussions in Sec 4.4 which addresses a few of the reviewer’s questions. There, we study sequences of 2 attacks including non-Lp starting attacks. These experiments make it clear that in finetuning ALR is much better compared to random (Uniform and Gaussian) regularization which hurts Union accuracy across the 2 attacks.

Theory

Thm 3.1, generalization and forgetting bounds

Thm 3.1 holds for any hypothesis $h$, so any model trained with the regularizer in the RHS will have a reduced loss gap. We find that this bound correlates with robust loss gaps in practice (Appendix E). Refining the bound to account for both generalization and forgetting is an interesting future direction. Our analysis relies on distances between representations for inputs to a fixed model rather than changes in representations through training. The latter must take into account changes in the representation space induced by training on new attacks, which is not well-understood. With Thm 3.1 and Cor 3.2, we aim to strike a balance between robustness against individual attacks (both seen and unseen), union accuracy, and accuracy on clean samples.

Generalizing theory to more than 2 attacks

Thm 3.1 and Cor 3.2 hold for any two attacks, whether or not they were seen during the course of training. For a larger set of attacks, the maximum loss gap between any attack pair is subject to the bound in Thm 3.1. Attacks can be defined to optimize over the union of multiple adversarial constraints, allowing us to extend our theoretical results to the union of multiple attacks.

Experimental

PGD step size

PGD is used only in adversarial training. Evaluations use AutoAttack, which adapts step size for accurate robustness assessment. Prior work (Gowal et al. 2020, Rice et al. 2020) also omits random initialization in L2 adversarial training.

Selection of the perturbation budget

Comparing attack strengths, especially non-Lp, is challenging. We use default budgets from the original attack papers. Developing comparative metrics for attack strengths is an interesting direction.

How to set the regularization strength?

See “How should we set the regularization strength parameter?” in our response to Reviewer 9HKR.

Model selection

We select the epoch with the highest average validation accuracy on known attacks, effectively performing optimal early stopping to avoid robust overfitting.

Average vs. Union Accuracy

The choice between them depends on application. Safety-critical settings prioritize union accuracy since it captures the worst case.

Other

TRADES comparison

TRADES is designed for improving clean accuracy tradeoff while ALR is designed for improving generalization across (seen and unforeseen) attacks. Since TRADES regularizer also maximizes a distance (KL instead of L2) in the logit space, we expect it can also improve generalization across attacks as well and provide results below. Similar to experiments with ALR, we regularize on top of PGD L2 and Linf adversarial training. Regularization strength in parentheses.

Notably, increasing TRADES strength in Linf training trades off Linf performance, whereas ALR does not.

Appendix results

Tables in App. H.1 mirror Table 3’s ablations on initial training, showing consistency across datasets and attacks. These do not involve finetuning; for finetuning results, see Fig 3. Discussion of Appendix figures is currently located early on before related figures. We will fix this in the camera-ready.

Single step optimization in ALR

ALR optimizes worst-case logit distance which we compute with a single PGD step. This is less precise than multiple steps but more efficient, aligning with our goal of improving efficiency.

Thank you for your positive appraisal of the paper and interesting questions.

What is the function of the regularization term ALR? When using it at the pretraining stage, does it accelerate the funetuning stage or make the initial model more robust to unknown attack? To prove this, it is better to add ablation study using AT and AT + ALR as init model to conduct the finetuning process.

ALR in the pretraining stage serves to improve the generalization to unforeseen attacks (discussed in the second paragraph of Section 4.2), which then provides a better starting point when finetuning the model to the new attack. In Figure 5 in the Appendix, we have provided a comparison between finetuning (without regularization) from an AT + ALR initial model and finetuning (without regularization) from an AT initial model.

When used in finetuning, ALR serves to reduce forgetting of robustness to previous attacks in the sequence, which we demonstrate in Table 1.

L0 Attack

Because the goal of CRT is to quickly adapt the model to unforeseen attacks when they become known to the defender, we chose to use the same set of attacks (Linf, L2, StAdv, ReColor) used for evaluation in works on unforeseen robustness [1,2] as well as incorporate attacks from a benchmark for unforeseen robustness (Gabor, Snow, Pixel, Kaleidoscope, Glitch, Elastic, JPEG, Wood) [3]. This is why we opted to use the attack set that we used in evaluation.

L0 attack involves combinatorial optimization and it is unclear whether it can be easily integrated into adversarial training which our framework is based off of. Additionally, [4] demonstrates that L0 attacks can be quite weak and needs to use large per-pixel perturbations.

[1] Laidlaw et al. (2021). Perceptual Adversarial Robustness: Defense Against Unseen Threat Models. International Conference on Learning Representations (ICLR).
[2] Dai, S., Mahloujifar, S., & Mittal, P. (2022). Formulating robustness against unforeseen attacks. Advances in Neural Information Processing Systems, 35, 8647-8661.
[3] Kaufmann, M., Kang, D., Sun, Y., Basart, S., Yin, X., Mazeika, M., ... & Hendrycks, D. (2019). Testing robustness against unforeseen adversaries. arXiv preprint arXiv:1908.08016.
[4] Zuo, F., Yang, B., Li, X., & Zeng, Q. (2019). Exploiting the inherent limitation of l0 adversarial examples. In 22nd International Symposium on Research in Attacks, Intrusions and Defenses (RAID 2019) (pp. 293-307).

Definition 2.1

The variables $t$ and $\delta_{\text{known}}$ are important for rigorously defining the problem that we are studying (continual adaptive robustness) and what goals we hope a good algorithm for this setting will achieve, which we connect to our proposed CRT in Section 3.1. Specifically, we become aware about the existence of attacks sequentially over time and at a specific point in time $t$, we would want to be robust against attacks that we have known for a while, have some robustness to recently introduced attacks, and recover quickly from new attacks. To model this, we have 3 problem parameters, (1) $\delta_{\text{known}}$ which specifies the loss threshold that we can tolerate on attacks that we’ve known for a while, (2) $\delta_{\text{unknown}}$ which specifies the loss threshold that we can tolerate on new recently introduced attacks, and (3) $\Delta t$ which specifies how long the model has to recover from new attacks. “Recovering from new attacks” basically means that threshold for tolerated loss switches from $\delta_{\text{unknown}}$ to $\delta_{\text{known}}$ with $\delta_{\text{known}} < \delta_{\text{unknown}}$.

These three quantities also serve to motivate the metrics we measure in the experimental section: Union (known) and Avg (known) correspond to $\delta_{\text{known}}$, Union (all) and Avg (all) give a sense of $\delta_{\text{unknown}}$ and the training time corresponds to $\Delta t$. These connections are discussed within the results discussion in Section 4.2.

In Table 1, the robust accuracy of Union(All) drops after the fine-tuning stage, does this mean the initial model is actually the best model against unknown adversarial attack?

This result suggests that the features used for classifying robustly under StAdv attack are different from those for Linf attack, so when we fine-tune the model on StAdv attack at time step 1, the generalization to Linf attack gets worse, which in turn impacts the union accuracy. The fine-tuned model's performance also depends on the attack sequence, so it is hard to conclude that the initial model will be better in terms of unforeseen robustness or not. For example, in Table 5 in the Appendix, union accuracy steadily increases after fine-tuning. In general, it is better to fine-tune on known attacks because these attacks are the ones that the defender is confident will affect the model’s performance.

Thank you for your insightful review and positive appraisal of our paper.

Discussion of gradual domain adaptation

Thank you for pointing us to this line of work. The work referenced studies shifts in data distribution over time and proposes gradual self-training to adapt the source model without access to labels. Meanwhile, we propose regularized CRT as a solution to the expanding space of attacks over time, with the data distribution itself remaining the same, with access to attacks and labels. We will add a discussion of this related direction into Appendix A.

Regularization on intermediate representations

We provide experiments with regularization on the features at the layer before the logits (Results in Appendix Table 4 rows labelled “+ALR feature”). Overall, we observe similar results compared to logit level regularization. We also provide theoretical results for regularization at the layer before the logits in Appendix C.3.

Comparing to methods from continual learning

Thank you for this suggestion. We experimented with using EWC for finetuning for StAdv robustness from an L2 robust model with the FT Single approach. We provide results for 3 different strengths (in parentheses) of EWC compared to unregularized FT Single and FT Single + ALR. Overall, we find that ALR’s improvement in robustness on known and unforeseen attacks is significant compared to EWC. EWC’s improvement over FT Single is similar to using FT Croce results in Table 1 (Time step 1) which uses replay of previous attacks in finetuning. We will add these comparisons into the Appendix.

Extension to scenarios where defender has limited knowledge about the attack type

This is an interesting direction. In our work, we focus on changes in the defender’s knowledge of attacks over time which is useful in cases such as a research or security team discovering a new attack type. A real-time attack setting poses new challenges:
No access to threat model- the defender does not know the threat model and cannot generate adversarial examples. They only have access to the perturbed data generated by the adversary.
Missing true labels and no access to the original unperturbed input - the defender also does not have the corresponding true labels or the original clean input for use in training.
Few shot updates - it becomes critical that the model can be made robust with only a few examples of successful attacks, otherwise it means that the adversary has been exploiting the vulnerabilities of the model for a long time Defending in this setting is outside of the scope of this paper, but potentially using generative models in order to model the perturbation [1] used by the adversary can help to bridge the gap from points (1) and (2) and allow for the defender to apply the attack on their own dataset and finetune with our proposed CRT + ALR. If the generative model is able to learn to model perturbations with only a few adversarial examples, then this can also address (3). We will add this discussion into Appendix B’s discussion of future directions.

[1] Wong et al. 2020 Learning perturbation sets for robust machine learning. ICLR

Visual examples of different attack types

Thank you for this suggestion, we will add visual examples of each attack into the Appendix.

Discussion of potential applications

Solving CAR is of interest in any safety-critical domain where an attacker is motivated to evade a ML model. A good example is automated content moderation, where malicious actors try to post content that violates policies by uploading obfuscated images [2]. Strategies naturally evolve over time for motivated attackers who can also use open-source methods proposed in the literature. As ML models will continue to be used in sensitive domains such as finance, cyber-physical systems and medicine, model deployers need methods to update their models to evolving threats. We will add this discussion to the updated paper.

[2] Stimberg et al. (2023) "Benchmarking robustness to adversarial image obfuscations." NeurIPS 2023

How should we set the regularization strength parameter?

We recommend selecting regularization strength based on how much tradeoff in clean accuracy (and starting attack accuracy in the case of uniform and gaussian regularization) that the model deployer can tolerate for the application.