AutoLoRa: An Automated Robust Fine-Tuning Framework

Many thanks for your comments! Please find our replies below.

[Reply to W1.1] TWINS requires expensive computation to grid-search the appropriate hyperparameters, which motivates us to automate the scheduler of hyper-parameters.

TWINS [1] is sensitive to hyper-parameters, such as the initial learning rate (LR) as shown in the following table.

Therefore, the authors of TWINS [1] conducted a time-consuming grid-search for hyper-parameters including the initial LR $\eta^{(0)} \in \\{3e-4,1e-3,3e-3,1e-2,3e-2\\}$, the weight decay $\mathrm{WD} \in \\{1e-5,1e-4,1e-3,1e-2\\}$, and the scalar $\gamma \in \\{ 0.1,0.2,0.3,0.4,0.5,1.0\\}$, which aimed to achieve the state-of-the-art (SOTA) performance. It means that TWINS needs to repeat $120$ ($= 5 \times 4 \times 6$) runs of fine-tuning in order to achieve the SOTA performance for each backbone and each downstream task, which is extremely time-consuming and inconvenient for practical usage.

By leveraging our proposed automatic scheduler, AutoLoRa can automatically achieve the SOTA performance by ONE run of robust fine-tuning on various models and various downstream tasks.

[Reply to W1.2] We make the first effort to propose an automated LR scheduler to avoid expensive grid-searching of the hyper-parameters.

In Table 5, TWINS w/o LR scheduler carefully selects the appropriate hyper-parameters for each downstream task via expensive grid-search. Table 5 shows that TWINS w/ the LR scheduler can achieve comparable performance to TWINS w/o the LR scheduler, which validates the effectiveness of the LR scheduler in avoiding grid-search.

[Reply to W2] We provide extensive results under various factors in Eq. (7).

We denote the factor as $\lambda_2^{\max} \geq 0$, i.e., $\lambda_2^{(e)} = \lambda_2^{\max} \cdot \mathrm{SA}(\mathcal{D}_{\mathrm{train}}; \\{\theta_1^{(e)}+B^{(e)}A^{(e)}, \theta_2^{(e)}\\})^\alpha$ (refer to Eq. (7)). There is a trade-off between standard accuracy and robust accuracy under AutoAttack (AA) by adjusting the factor $\lambda_2^{\max}$. We set $\lambda_2^{\max}=6.0$ by default to achieve good adversarial robustness without sacrificing too much standard generalization.

The experiments are conducted on CIFAR-100 using ResNet-18.

[Reply to W3] In Table 4, we report performance using different adversarially pre-trained FEs under $\epsilon_{\mathrm{pt}} \in \\{ 0, 1/255,2/255, 4/255, 8/255\\}$. In Table 2, we report performance using the adversarially pre-trained FE under $\epsilon_{\mathrm{pt}} = 4/255$, which is the same setting as TWINS.

$\epsilon_{\mathrm{pt}}$ refers to the adversarial budget used during the pre-training phase. We kept the adversarial budget as $8/255$ during the fine-tuning and evaluation phase. You can find that the results of $\epsilon_{\mathrm{pt}} = 4/255$ in Table 4 are consistent with the results in Table 2.

[Reply to W4] Thanks for pointing out this typo! We have revised it in our manuscript.

[Reply to W5] We provide extensive results on Wide ResNet of depth 28 and width 10 (WRN-28-10), which validate that AutoLoRa is compatible with various models.

We could not provide the results of TWINS because the authors of TWINS [1] did not implement TWINS on other models except for ResNet. We used the pre-trained WRN-28-10 on ImageNet-1K provided by [2] and applied Vanilla RFT and AutoLoRa on the CIFAR-100 task. RA refers to the robust test accuracy under PGD-20.

---

[Update extensive results for mitigating W5] Besides existing experiments on ResNet-18, ResNet-50, and WRN-28-10, we provide results on extra pre-trained models, i.e., Vision Transformer (ViT) and Data-Efficient Image Transformers (DeiT) to validate the effectiveness of AutoLoRa on various model structures.

1. ViT (S/16)

2. ViT (B/16)

3. DeiT (DeiT-Tiny)

4. DeiT (DeiT-small)

---

References:

[1] TWINS: A Fine-Tuning Framework for Improved Transferability of Adversarial Robustness and Generalization, CVPR 2023.

[2] Using Pre-Training Can Improve Model Robustness and Uncertainty, ICML 2019.

[Reply to Q1] We demonstrate the results of ''Vanilla RFT + scalar scheduling'' and "TWINS + scalar scheduling" as follows.

The results validate that the scalar scheduler is applicable to Vanilla RFT and TWINS. Besides, it justifies the effectiveness of the auxiliary LoRa branch in enhancing performance.

The experiments are conducted on CIFAR-100 using ResNet-18. Following your suggestion, the scalar scheduler for $\lambda_2$ is applied to $\beta$.

[Reply to Q2] We provide a detailed illustration of calculating gradient similarity as follows.

At each epoch, for each convolutional layer of the FE, we obtain a gradient similarity value according to Eq. (3) and (4). Then, we demonstrate the average and the standard deviation of all the convolutional layers' gradient similarity values in the FE at each epoch in Figure 1(a).

[Reply to Minor Comments] Thanks for your suggestion! We mention the relationship in Appendix A.1.

References:

[1] Theoretically Principled Trade-off between Robustness and Accuracy, ICML 2019.

[2] Understanding and Mitigating the Tradeoff Between Robustness and Accuracy, ICML 2020.

Many thanks for your comments! Please find our replies below.

[Reply to W1] We provide extensive results of AutoLoRa with or without the automated scheduler.

By comparing Row 1 with Row 2 in the above table, we can find that the automated scalar scheduler is beneficial to performance improvement and avoid the expensive grid-search.

By comparing Row 1 with Row 3, we can find that the automated LR scheduler can help obtain the satisfied performance without expensive grid-searching for hyper-parameters.

The above table uses ResNet-18 on the CIFAR-100 dataset. Without the automated scalar scheduler, we set $\lambda_1=0.5$ and $\lambda_2=6.0$. Without the automated LR scheduler, we set the initial learning rate as 0.01 and the learning rate is divided by 10 and 100 at Epoch 30 and 50.

[Reply to W2.1] We detailed the issue of sensitivity to hyper-parameters as follows.

According to TWINS, Vanilla RFT and TWINS are senstive to the initial learning rate (LR), the weight decay, and the scalars. Therefore, the authors of TWINS conducted an expensive grid search on the aforementioned three hyper-parameters for Vanilla RFT and TWINS to obtain the best performance.

The following results show that Vanilla RFT and TWINS are sensitive to the initial LR caused by poor training stability. For example, when the initial LR is sampled from {0.001, 0.1, 0.03}, the robust accuracy (AA) ranges from 18.29 to 24.07 obtained by Vanilla RFT and 1.00 to 25.45 obtained by TWINS.

The experiments are conducted on CIFAR-100 using ResNet-18. The LR is divided by 10 and 100 at Epoch 30 and 50, following TWINS. Note that, TWINS fails to converge when the initial LR is 0.03.

[Reply to W2.2] We empirically validate that AutoLoRa mitigates the issue of sensitivity to hyper-parameters (e.g., the initial LR).

We can find that AutoLoRa is less sensitive to hyper-parameters such as the initial LR since AutoLoRa's standard deviation of the accuracy among different initial LRs is very small compared to Vanilla RFT and TWINS.

According to the table in [Reply to W2.1], we show the standard deviation of the accuracy among different initial LRs in the above table.

[Reply to W3] The trade-off between robust and standard test accuracy could be mitigated by utilizing larger models. Further, we figure out a hyper-parameter (i.e., $\lambda_2^{\max}$ in Eq. (7)) that can adjust this trade-off.

The trade-off between accuracy and adversarial robustness ubiquitously exists [1,2]. By comparing the results on CIFAR-10 and CIFAR-100 in Tables 1 and 2, we can find that utilizing larger models (that contain more trainable parameters) could mitigate the trade-off since both standard and robust accuracy get improved using ResNet-50.

Further, we find that the hyper-parameter $\lambda_2^{\max}$ in Eq. (7), which is used for controlling the scale $\lambda_2^{(e)}$, can adjust the trade-off. We set $\lambda_2^{\max}=6.0$ by default in our paper. The following table shows that an appropriate $\lambda_2^{\max}$ for AutoLoRa (e.g., $\lambda_2^{\max}=3.0$) can yield improvement on both the standard and robust accuracy compared to TWINS.

The experiments are conducted on CIFAR-100 using ResNet-18.

Many thanks for your comments! Please find our replies below.

[Reply to W1] AutoLoRa is applicable to fine-tuning the pre-trained models via self-supervisions.

We applied AutoLoRa to finetune the pre-trained models via robust self-supervision including DynACL [1] and DynACL-AIR [2]. The pre-training and fine-tuning were conducted on CIFAR-10 and STL-10 datasets, respectively. We report the performance in the following table. Here, the robust accuracy is evaluated by AutoAttack.

The results validate that AutoLoRa can improve both standard and robust test accuracy of the pre-trained models via self-supervisions in the downstream task.

[Reply to W2] AutoLoRa achieves better robustness under various attacks than Vanilla RFT and TWINS.

We evaluated the robustness under three strong white-box attacks (APGD-CE [3], APGD-DLR [3] and FAB [4]) and one strong black-box attack (i.e., Square Attack [5]). We evaluate the robustness on CIFAR-10 of the pre-trained ResNet-18 after finetuning in the following table.

The experiments are conducted on CIFAR-10 using ResNet-18.

References:

[1] Rethinking the Effect of Data Augmentation in Adversarial Contrastive Learning, ICLR 2023.

[2] Enhancing Adversarial Contrastive Learning via Adversarial Invariant Regularization, NeurIPS 2023.

[3] Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks, ICML 2020.

[4] Minimally distorted adversarial examples with a fast adaptive boundary attack, ICML 2020.

[5] Square attack: a query-efficient black-box adversarial attack via random search, ECCV 2020.

Many thanks for your comments! Please find our replies below.

[Reply to W1&Q1.2] The second cross-entropy (CE) term $\ell_{\mathrm{CE}}(h_{\theta}(\tilde{x},y))$ aims to provide ground-truth labels for the FE to learn the adversarial data, which can improve the adversarial robustness as validated in the following table.

Without the second CE term, the FE would be only regulated by the KL term that uses the natural soft labels generated by the LoRa branch. However, the natural soft labels could contain wrongly classified labels, which could degrade the performance. Empirically, we found that the natural training error evaluated on the LoRa branch keeps about $30\\%$ on CIFAR-100, which indicates that there are some noisy labels in natural soft labels generated by the LoRa branch. Thus, it necessitates the second CE term to provide the ground-truth labels for updating the FE together, which can further improve performance.

The following empirical results validate that with the second CE term, both adversarial robustness and standard generalization achieved by AutoLoRa get improved. The experiments are conducted on CIFAR-100 using ResNet-18.

[Reply to W2&Q1.1] AutoLoRa mitigates the issue of divergent gradient directions, which is further empirically validated in Figure 3 (in Appendix A.4).

Figure 3 shows that AutoLoRa achieves high cosine similarity between the two terms of the adversarial objective in Eq. (5), which empirically justifies that AutoLoRa mitigates divergence.

Note that both the second CE term and the KL term aim to let the FE learn the adversarial data. In this way, the LoRa branch disentangles the model to learn the adversarial and natural data via the FE and the LoRa branch, respectively. Thus, AutoLoRa can mitigate the issue of divergent gradient directions.

[Reply to Q2] The term parameter-free refers to that you do not need to specify any hyper-parameters (e.g., the initial learning rate, the learning rate scheduler, the scalars) when applying AutoLoRa to various downstream tasks (e.g., CIFAR, DTD-57, CUB-200) and various models (e.g., ResNet-18, ResNet-50).

The term parameter-free is inspired by the title of AutoAttack [1], in which parameter-free refers to that you do not need to specify any hyper-parameters while utilizing the AutoAttack to attack various models on various datasets.

AutoLoRa is parameter-free benefiting from our proposed automated scheduler for the learning rate and the scalars. We empirically validate that AutoLoRa is parameter-free since AutoLoRa only needs to specify the downstream dataset and the model, without specifying any other hyper-parameters, to automatically obtain superior performance in downstream tasks.

References:

[1] Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks, ICML 2020.

First, many thanks for your commending that our heuristic scheduler is good and makes a good contribution!

Second, following your suggestions, we provide results on extensive model structures and extensive downstream datasets to strengthen our parameter-free claim.

Besides existing experiments on ResNet-18 and ResNet-50, we provide results on three extra pre-trained models including Wide ResNet (WRN), Vision Transformer (ViT), and Data-Efficient Image Transformers (DeiT) to strengthen our parameter-free claim.

1. WRN with depth 28 and width 10

2. ViT (S/16)

3. DeiT (DeiT-tiny)

4. DeiT (DeiT-small)

---

[Update extensive results]

5. ViT (B/16)

---

As for WRN, pre-training and fine-tuning are conducted on ImageNet-1K and CIFAR-100, respecitve. As for ViT and DeiT,pre-training and fine-tuning are conducted on ImageNet and CIFAR-10, respecitve. We do not provide the results of TWINS since TWINS is only implemented on ResNet.

Besides existing experiments on CIFAR-10, CIFAR-100, DTD-57, DOG-120, CUB-200, and Caltech-256, we provide results on three extra downstream dataests including STL-10, Caltech-101, and Cars-196 to strengthen our parameter-free claim.

1. STL-10

2. Caltech-101

3. Cars-196

As for STL-10, we used pre-trained ResNet-18 via adversarial contrastive learning on CIFAR-10 of $32 \times 32$ resolution and then fine-tuned the model on STL-10 of $32 \times 32$ resolution. We do not provide the results of TWINS since TWINS is only implemented on datasets of $224 \times 224$ resolution. As for Caltech-101 and Cars-196, we used ImageNet-1k pre-trained ResNet-50 for fine-tuning.