We extend our appreciation for your constructive feedback on our manuscript. Below, we address each of your points comprehensively. Should there be any additional queries or clarifications needed, please feel free to let us know.

Q1. Adversarial robustness evaluation of unseen attacks

• We appreciate the reviewer's invaluable suggestion.
  • Our claim that "AMT boosts the adversarial generalization" is mainly grounded in our results in Table 3 where adversarial robustness is evaluated under distribution shifts using the same meta-tuned LoRAPool on the source domain ImageNet.
  • We conducted additional experiments to measure the adversarial robustness generalization to unseen threat models, including $\ell_{\infty}$- and unseen $\ell_2$-bounded attacks with different perturbation budgets $\epsilon$. For each dataset, we sample 600 $5$-way $1$-shot tasks and generate adversarial examples using 10 steps of PGD with the step size $\epsilon/10$ for $\ell_{\infty}$ and $\epsilon/8.5$ for $\ell_2$ attacks, respectively. The results, shown in Table 10 of the rebuttal PDF, demonstrate our method AMT significantly enhance adversarial robustness against unseen attacks under distribution shifts for pre-trained vision transformer. Also, compared to previous adversarial few-shot learning methods StyleAdv, our AMT does not sacrifice in-domain generalization.

Q2. Adversarial robustness evaluation of AutoAttack under distribution shifts

• We appreciate this great suggestion, and in response, we have incorporated additional experiments to measure adversarial robustness against AutoAttack [1] under distribution shifts. Specifically, we ground our method and the baseline on the adversarially pre-trained ViT-Small [2] and use APGD with cross-entropy and targeted DLR loss, FAB-attack and the Square Attack to generate adversarial examples on the 100 sampled $5$-way $1$-shot tasks on each dataset. The $\ell_{\infty}$-bounded perturbation at the radius $\epsilon_{\infty}=4 / 255$ is adopted. The results, shown in the below table, demonstrate our method AMT consistently boosts adversarial generalization even under stronger AutoAttack in terms of both in-domain and out-of-domain robust accuracy.

Method	ImageNet	Omniglot	Aircraft	CUB	DTD	QuickDraw	Fungi	Flower	Traffic Sign	MSCOCO	Avg.
PM	29.36	36.52	3.88	15.06	14.20	29.80	3.61	20.48	10.26	8.26	17.14
AMT	39.96	61.48	8.88	24.04	23.12	51.48	11.09	44.76	23.20	22.00	31.00

[1] Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, 2020

[2] Revisiting Adversarial Training for ImageNet: Architectures, Training and Generalization across Threat Models. In NeurIPS, 2023.

Q3: Hyper-parameter Study of the LoRA rank and the pool size

• We conduct the ablation analysis following the reviewer's valuable suggestion to investigate the impact of the rank of LoRA and the size of LoRAPool. We also report the mean and variance of perturbation budget candidates $\epsilon$ during adversarial meta-tuning. The results, as illustrated in Tables 5 and 7 of the rebuttal PDF, indicate that (1) our model is not very sensitive to the rank of LoRA; (2) a sufficiently diverse but the largest pool leads to improvements in performance.

We appreciate your constructive comments on our paper. Please kindly find our response to your comments below. We hope that our response satisfactorily addresses the issues you raised. Please feel free to let us know if you have any additional concerns or questions.

Q1. More discussions concerning other PEFT techniques during meta-tuning

• We sincerely thank the reviewer for the constructive comments and will incorporate the following discussion into our revised manuscript.
  • We have conducted additional experiments to adversarially meta-tune full parameters [3], FiLM [1] (after LN layers since there are no BN layers in ViT), Adapter [2], and LoRA. The attack budget is randomly sampled for each training task from the candidate pool which is the same as AMT. The results, shown in the table below, demonstrate that the performance of Adapter and LoRA is comparable in the context of adversarial meta-tuning and outperforms full or FiLM-based meta-tuning.
  • We would like to highlight that compared with FLUTE which combines multiple FiLMs with a parametric classifier as the initialization for FiLM-based test-time fine-tuning, our LoRAPool with non-parametric merging mechanism adaptively integrates the LoRAPool into pre-trained weights, and thus provides better flexibility and compatibility with advanced test-time fine-tuning techniques to further improve the few-shot learning performances. For example, Tabel 1 in the work [4] has demonstrated that both LoRA- and Adapter-based test-time fine-tuning are not optimal choices for vision transformers.
  • We are still pending for the results of FLUTE-style test-time fine-tuning, which we will supplement as soon as possible during the discussion period.

Adversarial Meta-tuning	Test-time merge	Test-time fine-tuning	ImageNet	Omniglot	Aircraft	CUB	DTD	QuickDraw	Fungi	Flower	Traffic Sign	MSCOCO	Avg.
Full	-	-	64.31	62.81	38.46	76.23	60.42	57.99	56.31	81.80	57.31	54.22	60.98
Single FiLM	-	-	63.23	63.41	37.67	74.41	59.29	57.60	55.23	80.05	58.86	54.57	60.43
Single Adapter	-	-	64.68	65.32	38.43	75.37	59.68	58.35	55.90	81.69	58.31	54.05	61.18
Single LoRA	-	-	63.91	65.05	39.44	76.95	58.46	58.35	56.39	82.29	59.56	53.69	61.41
FiLM Pool	classifier	FiLM
Adapter Pool	classifier	LoRA
LoRAPool	classifier	-	67.22	64.60	37.99	77.96	62.65	57.11	56.62	80.23	58.36	56.10	61.89
LoRAPool	criteria	-	68.80	71.95	42.90	79.95	62.99	59.62	59.06	85.37	63.78	57.14	65.16
LoRAPool	criteria	LoRA
LoRAPool	criteria	PMF [3]	68.80	77.83	42.90	79.95	63.77	63.72	59.06	85.37	63.87	57.37	66.26
LoRAPool	criteria	ATTNSCALE [4]	68.80	79.43	42.90	79.95	63.08	65.66	59.06	85.37	64.13	58.24	66.66

[1] Learning a Universal Template for Few-shot Dataset Generalization. In ICML, 2021

[2] Adaptformer: Adapting vision transformers for scalable visual recognition. In NeurIPS, 2022

[3] Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make a difference. In CVPR, 2022

[4] Strong baselines for parameter-efficient few-shot fine-tuning. In AAAI, 2024

Q2. More discussions concerning other data augmentation techniques

• Thank you for your insightful suggestion. We have conducted additional comparisons for AMT against other data augmentation methods, including ALT [1] and Rand Conv [2], as suggested.
  • Our experiments were conducted under fair conditions using a single LoRA (pool size $P=1$) during meta-tuning for all methods. Specifically, following ALT [1], we employed a learnable adversarial transformation network consisting of 5 convolutional layers with a kernel size of 3 and LeakyReLU activation. The adversarial learning rate was set to $5 \times 10^{-5}$, with 10 adversarial steps. For the method employing an attack candidate pool, we randomly select the attack budget from candidates for each training task, with $\epsilon$ values of {8/255, 6/255, 0.1/255, 0.01/255} for our method, and step number of {1, 3, 5, 10} for ALT.
  • The results in Table 9 of the rebuttal PDF demonstrate that static data augmentation cannot effectively simulate the large domain shift required for robust generalization across diverse datasets (e.g., Omniglot). Our AMT with standard pixel-level adversarial attack achieves comparable or superior generalization improvements for pre-trained vision transformers across OOD tasks.

Q3. Other data augmentation techniques deserve future work

We sincerely appreciate the reviewer's invaluable suggestion. We would like to highlight that we present a framework by constructing the robust LoRAPool with test-time merging to significantly boost the robust generalization of the pre-trained vision transformer. In this context, we use adversarial attacks, characterized by the size of the perturbation budget, as an example to mimic different distributional shifts to construct different LoRAs. Our experiments demonstrate the effectiveness of using adversarial training. The focus and contribution of this paper is the whole framework of the algorithm instead of the optimal data augmentations. In addition, we believe the data augmentation to achieve the optimal performance also depends on the OOD test set we use. Nevertheless, we agree data augmentation technique is important for this problem and leave investigating better training data augment techniques (e.g., Virtual Outlier Synthesis (VOS)) as future works.

Thank you sincerely for your thoughtful feedback on our work. Below, we have provided a detailed explanation for your concerns as follows. Please do not hesitate to let us know if you have any further questions.

Q1. Technical contributions concerning adversarial singular value and vector perturbation and robust LoRAPool

• We acknowledge the reviewer's scrutiny of our technical contributions and would like to clarify that our work is definitely not a simple summation of existing techniques. Different components of our methods are closely related and adapted to our goal: to boost the robust generalization of pre-trained models in out-of-domain few-shot learning.
  • Specifically, inspired by the low-rank strategies when training transformers, we do not directly perturb the weight matrices as done in SAM, instead our perturbations are in the spectral space: we perturb the singular values and the singular vectors to boost the performance.
  • In addition, we use different adversarial perturbations (either different types or different magnitudes) to generate a LoRA pool consisting of multiple low-rank structures and use adaptive merging to construct the best-adapted parameters for different out-of-distribution tasks during the test time. We acknowledge that both SAM and LoRA motivate our solution, but we need to clarify that our algorithm is significantly different from vanilla SAM or LoRA and much better adapted to the problem we aim to solve. Below are additional comparisons to validate the effectiveness of our novel solutions.
    • The results, reported in Table 2 of the rebuttal PDF, demonstrate that our method outperforms SAM by 1.56% on average.
    • As shown in Table 1 of the rebuttal PDF, the robust LoRAPool with perturbation-specific parameters effectively avoids interference between attacks and significantly enhances the OOD generalization without ID compromise. For uniform strategy, we adopt the average attack strength ($\epsilon=3.5$) of candidate configurations and meta-tune 4 LoRAs with different seeds. The random strategy means we randomly sample one attack budget $\epsilon$ for each training task from the same attack candidate configurations.
    • We also humbly highlight that our contribution of singular value trimming and non-parametric test-time merging mechanism is also novel and effective for few-shot learning, as supported by the results in Tables 4 and 5 of the main paper.

Q2. Clarification of the top_k operation

We apologize for any potential misunderstandings. The operation top_k before softmax in Eq. (7) refers to selecting the top $k$ LoRA modules with the largest score of $-\beta (1- (\lambda C-(1-\lambda)V)$ and the rest LoRAs are deactivated for the current task.

Q3. Clarification of the tuning-free setting and test-time fine-tuning setting

We apologize for any potential misunderstandings.

• The tuning-free setting does not involve additional training on the support set. We adaptively merge meta-tuned LoRA into pre-trained models via the formula in line 206 and perform prototype-based classification. Aside from this, the test-time fine-tuning setting allows for training on the support set according to different fine-tuning methods such as fine-tuning full parameters (PMF) or partial parameters (ATTNSCALE)
• We evaluate under both settings to demonstrate AMT's (1) effectiveness in learning a well-generalized initialization for pre-trained models even without time-consuming fine-tuning, and (2) compatibility with advanced fine-tuning techniques to further improve the few-shot learning performances.

Q4. Clarification of variant removing the adversarial perturbations on singular values and vectors

We appreciate the reviewer's feedback. When removing the adversarial perturbations on singular values and vectors, we inject the worst-case perturbations into the input only, without perturbing the singular value and vectors of LoRA. Other configurations remain the same as the final version of AMT. We would like to humbly clarify that this adjustment does not imply not using SAM.

Q5. Clarification of evaluated model

We appreciate the reviewer's invaluable feedback and we clarify that all results presented in Table 1, 2 and 3 are achieved simultaneously by the same model using a meta-tuned robust LoRAPool on the source domain ImageNet.

• We would like to highlight that thanks to the diverse design of the adversarial LoRAPool, our AMT improves the trade-offs between adversarial robustness and clean accuracy, as well as between ID and OOD generalization, as supported by the results compared to the clean meta-tuning method (PMF) and adversarial few-shot learning method (StyleAdv) in Table 1, 2 and 3.
• We conducted additional experiments, employing different $\lambda_{adv}$. The results, in Table 3 of rebuttal PDF, demonstrate that we can use $\lambda_{adv}$ to adjust the preference of our robust LoRAPool to either clean or adversarial environments.

Q6. Ablation studies to support the improvement of perturbation on singular values and vectors

We appreciate the reviewer's insightful suggestion and conducted additional comparisons, pitting AMT against original LoRA initialization, for which we incorporated adversarial perturbations in the weight space. The results in Table 4 of rebuttal PDF demonstrate that AMT achieves superior performance, highlighting the effectiveness of our adversarial singular value and vector perturbation in boosting the model's generalization capability.

Q7. Hyper-parameter study

• We have conducted supplementary experiments. The results, as illustrated in Tables 5, 6, 7, and 8 of the rebuttal PDF, indicate that (1) our model is not very sensitive to the rank of LoRA and the number of attack steps; (2) a sufficiently diverse but large pool leads to improvements in performance. The results also justify our choice of top-2.

We sincerely thank the reviewer for providing valuable feedback. We detail our response below point by point. Please kindly let us know whether you have any further concerns.

Q1. Instances of novel and adversarial environments

We appreciate the reviewer's great feedback and agree that discussion of instances of adversarial environments in real-life applications will strengthen the motivation of this work. We will incorporate the discussions and the related literature into our revised manuscript.

• Instances of novel environments: In real-world deployments, deep learning models in both medical and self-driving domains often encounter novel environments and suffer from distribution shifts between training and deployment data, including unseen pathologies [1], variations in hospital equipment and protocols [1], and diverse urban road scenarios [2].
• Instances of vulnerability to adversarial attacks: Furthermore, deep neural networks are vulnerable to physical-world adversarial attacks, which can lead to harmful diagnoses or unsafe driving decisions. For instance, adversaries can perturb sensor signals to deceive 2D or 3D medical imaging models [3, 4], manipulate traffic signs with malicious stickers to mislead autopilot systems [5], or fool the autopilot into following fake lane lines[6] or making unsafe trajectories [7].

[1] Understanding silent failures in medical image classification. In MICAAI, 2023.

[2] Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.

[3] Self-adaptive adversarial training for robust medical segmentation. In MICAAI, 2023.

[4] Adversarial attacks and defenses on AI in medical imaging informatics: A survey. In Expert Systems with Applications, 2022.

[5] Robust Physical-World Attacks on Deep Learning Visual Classification. In CVPR, 2018.

[6] Dirty road can attack: Security of deep learning based automated lane centering under physical-world attack. In USENIX Security, 2021

[7] On Adversarial Robustness of Trajectory Prediction for Autonomous Vehicles. In CVPR, 2022.

Q2. Clarification of writing

We apologize for the ambiguity and thank the reviewer for valuable feedback on our manuscript. We have carefully considered each of the points and propose the following revisions to enhance the coherence and readability of our paper:

• a) Connection with the previous sentence (line 15): We will clarify the transition between sentences. Specifically, we will revise the first two sentences as follows: "Building upon the capabilities demonstrated by large-scale pre-trained vision transformers in zero-shot scenarios, a few annotated examples can be leveraged to further enhance generalization capability, achieving impressive performance across a broad spectrum of downstream tasks."

• b) Transition in Paragraph (line 21): We acknowledge the abruptness of the transition and the unnecessary use of "However." We will revise as follows: "While few studies have explored how meta-tuning can maintain high performance under these conditions at the same time, it is crucial for real-world applications..."

• c) Coherence in Third Paragraph (lines 29, 31, 33): We will revise these sentences to ensure a more cohesive flow of ideas.

• d) Clarification of "double strongest perturbation" (line 45): We inject the strongest perturbations twice by initially attacking the input and subsequently perturbing the singular values and vectors with adversarial examples.

• e) Simplification of the sentence (lines 130-132): We will simplify the sentence to "To robustify the learned meta-knowledge, adversarial meta-tuning injects the worst-case adversarial perturbation $\delta$ to the input $x$."