Learning Transferable Robust Representations for Few-shot Learning via Multi-view Consistency

Weakness 1: Few-shot learning to meta-train on small-based datasets is no longer meaningful research.

• We respectfully contest this claim as it overlooks the relevance of such research in scenarios with computational or data constraints.
• While we recognize the impressive performance of large vision models like CLIP and MoCo, it is important to note that not all real-world problems afford the luxury of extensive computational resources and large datasets. For example, in the clinic problem or individual personal services, one might only have a limited number of samples for each class for diagnosis prediction or personal recognition and simultaneously need to safeguard against the threats of attacks.
• Consequently, a large number of data for each class is not a universally applicable or feasible approach for all real-world applications.
• Existing works [1, 2] also emphasize the importance of achieving generalization with scarce meta-training tasks in real-world applications.
• We politely request the reviewer to reconsider the significance of our work and research area in light of these considerations.

[1] Meta-Learning with Fewer Tasks through Task Interpolation, ICLR'22
[2] Adversarial Task Up-sampling for Meta-learning, NeurIPS'22

---

Weakness 2: The clean performance is poorer than existing clean meta-learning which costs more.

• While our clean performance is slightly lower than the clean meta-learning approaches (around -10%), the expense is justified by significant gains in robustness (around +40%). Furthermore, existing clean meta-learning models could not withstand adversarial attacks (shows 0% performance). This demonstrates the importance and critical need for adversarial training in diverse tasks and domains.
• We appreciate your concerns regarding the trade-off between clean performance and robustness against adversarial attacks. However, as shown in Table 1 (also summarized in the following), our approach achieves the highest robustness while having a minimal compromised clean performance compared to previous adversarial meta-learning. This demonstrates that the enhanced model’s meta-generalizability stems from our novel multi-view learning framework as the Reviewer rEWc acknowledges.

Weakness 3:Robustness is mainly derived from the use of PGD.

• There is a fundamental misconception that our robustness relies solely on the use of PGD. Our approach derives its transferable robustness from employing multi-view adversarial attacks on bootstrapped encoders.
• In particular, our approach addresses the limitation of a previous simple combination of PGD with a meta-learning framework, which failed to demonstrate transferable adversarial robustness. (Section 3.3: green text)
• We already experimentally demonstrated that existing AML approaches fail to achieve transferable robustness on unseen domains, where the best AML baseline (AQ) only shows an average of robustness 5.99% while ours achieves a remarkable average robustness of 28.20% (Table 1).
• Additionally, previous AML methods possess highly overlapped representation space on the novel domain, leading to poor transferable robustness. In contrast, ours represents well-discriminated representations even without training, supporting the importance of our multi-view representation learning (Figure 3).
• Moreover, when simply combining the PGD with recent clean meta-learning, those exhibit significantly less robustness compared to ours. (Table 3, also summarized in the following) |Method|Avg unseen clean|Avg unseen rob.| |:-|:-|:-:| |MAML+PGD|33.34|7.10| |MetaOptNet+PGD|45.54|12.61| |ProtoNet+PGD|41.96|5.68| |Ours|50.32|28.20|

---

Weakness 4: MetaOptNet could not perform better than the MAML-based approach.

• There is a critical misunderstanding of our work. We propose a multi-view latent attack that enables MAML to outperform a mere combination of PGD with MetaOptNet.
• As indicated in Table 3, when we simply apply PGD attacks to both MAML and MetaOptNet, MetaOptNet demonstrates superior performance across 5 different unseen domains, with a margin of 12.2% in clean and 5.51% in robust average accuracy, respectively.
• Nevertheless, our approach surpasses the simple combination of PGD and MetaOptNet, which distinctly evidences the contributions of our method.
• Additionally, the MAML approach can outperform MetaOptNet in certain stable settings, as demonstrated in [1]. This could further support the superior performance of our MAML-based approach over the simple combination of MetaOptNet in adversarial meta-learning tasks.

[1] How to Train Your MAML to Excel in Few-Shot Classification, ICLR'22

Dear Reviewer,

This is a gentle reminder that we have only 3 days remaining for discussions. Therefore, we kindly request you review our responses and the revisions. We have tackled all of the concerns you raised and have included additional experimental results in our previous responses.

For your convenience, we provide short summary of our previous responses below.
Thanks again for your time and effort in reviewing our work.

Best,
Author

---

Summary

W1. Few-shot learning is no longer a meaningful research area.

• The use of a large volume of data for each class is not universally applicable or feasible for all real-world applications, underscoring the continued importance of few-shot learning, as acknowledged in existing works.

W2. The clean performance is poorer than existing clean meta-learning which costs more.

• Our approach achieves superior robustness while having a minimal compromised clean performance compared to the previous adversarial meta-learning (AML) approaches.
• Compared to clean meta-learning, our method shows slightly lower clean performance (around -10%) while achieving significant gains in robustness (around +40%). Furthermore, existing clean meta-learning models could not withstand adversarial attacks, showing 0% performance.

W3. Robustness is mainly derived from the use of PGD.

• This is a misunderstanding of our work.
• Our transferable robustness stems from our multi-view adversarial attacks on bootstrapped encoders.
• A previous simple combination of PGD with meta-learning demonstrated inferior transferable robustness as shown in our manuscript (Table 1 and 3, Figure 3). Specifically, a simple combination of PGD and meta-learning (AML approaches) shows an average of robustness 5.99% while ours achieves a remarkable average robustness of 28.20%

W4. MetaOptNet could not perform better than the MAML-based approach.

• It is logical that when Approach A (Ours) is significantly more effective than Approach B (previous AML) (A>B), MAML+A could outperform MetaOptNet+B. This result clearly demonstrates the effectiveness of our approach.

MAML + PGD (Approach B) < MetaOptNet + PGD (Approach B) < MAML + Ours (Approach A)

• Moreover, the MAML approach can perform better than MetaOptNet in certain stable settings as demonstrated in the previous works.

Weakness 5: Recent work seems better, and needs comparison.

• Thank you for suggesting relevant baselines for our work. We will also add [1] to the main paper.
• However, it is important to note that the evaluation setting in [1] differs significantly from ours. Primarily, [1] evaluated the performance against FGSM attackers with a smaller size of epsilon of 0.01 which is weaker than our PGD-20 attacks with epsilon of 0.03.
• When we set the evaluating setting as [1] in CIFAR-FS, as shown in the following table, our approach achieves better performance than [1].
• Since [1] did not provide any codes for the model or dataloader, we will reproduce those results in our setting to add to Table 1.
• Our approach achieves 15% higher clean accuracy and 4% better robustness than [1] in CIFAR-FS.

Weakness 6: Evaluation against diverse attacks is recommended.**

• Thank you for the suggestions. We evaluated our models against SPSA [2], CW [3], and FGSM [4] attacks as shown in the following Table. We will add these results to our manuscript.
• Our approach could consistently demonstrate robust results against diverse attacks compared to the best-performing AML baseline (AQ).

Weakness 7: Variance for the experiments

• Thank you for the suggestion, we will add variances in our results.
• The variances of converged adversarial trained networks are not very large as shown in the following Table.

[1] Defensive Few-shot Learning. TPAMI 2022
[2] Uesato et al., Adversarial Risk and the Dangers of Evaluating Against Weak Attacks, PMLR 2018
[3] Uesato et al., Adversarial Risk and the Dangers of Evaluating Against Weak Attacks
[4] Gubri et al., LGV: Boosting Adversarial Example Transferability from Large Geometric Vicinity, ECCV 2022

Thank you for your valuable comments and thoughtful review, which included positive feedback highlighting the well-organized manuscript, reasonable motivations, and our novel idea.

---

In response, we have made every effort to address all the concerns you raised regarding our work. Please find our detailed responses below. If there are further concerns about our work, please do not hesitate to share them. We are more than happy to address any additional questions or concerns you may have.

---

Weakness 1-1: TRADES needs to be a straightforward baseline in the main experiment. In addition, some latest generic adversarial training methods are also recommended to be added to Table 1.

• Our tasks are few-shot meta-learning tasks that can not be easily achievable with vanilla supervised training, such as TRADES or generic adversarial training methods.
• However, we trained the model on a few-shot task of CIFAR-FS with TRADES as you requested. As shown in the following table, generic adversarial training, TRADES, only overfits the training tasks while unable to generalize to unseen few-shot tasks, since these models were not meta-trained.

Weakness 1-2: Other adversarial meta-learning approaches with the TRADES are needed.

• Thank you for your suggestions. We will add these results to our manuscript
• We trained the previous adversarial meta-learning approach using TRADES, as suggested by the reviewer.
• The table below illustrates that our method continues to outperform the integration of TRADES with prior adversarial meta-learning approaches. We will include these results in our manuscript.

Weakness 2: How to use two encoders in test time?

• There seems to be a misunderstanding regarding how our bootstrapped encoders are employed during the training. Our encoders operate as bootstrapped parameters during the inner-adaptation that are shared parameters. (page 5: algorithm last line/ page 5: green text)
• During the meta-optimization phase, we update only a single encoder. Consequently, a single encoder is employed at the test time. (page 6: green text)

Weakness 3: What is FSL classifier?

• Our FSL classifier is a fully connected layer as follows the setting used in [1].
• To reflect your comments, we added experimental details in the paper (page 7: orange text)

[1] Li et al., Meta-SGD: Learning to Learn Quickly for Few-Shot Learning

Weakness 4: Experiment comparisons are not fair.

• Due to the page constraints, we compared our method only with the most robust adversarial baselines, namely AQ.
• As shown in the following table, other AML methods also demonstrate inferior transferability in non-RGB datasets. Based on your feedback, we also added these results to the main paper (page 7: orange)

• We acknowledge the trade-off issue inherent in adversarial training. Nonetheless, as shown in Table 1 our method achieves superior robustness and the highest clean accuracy compared to other previous adversarial meta-learning approaches. Moreover, compared to the clean meta-learning method (MAML), our approach has an average of 3.51% and 0.94% less clean performance while achieving 23.96% and 20.73% robustness gains for CIFAR-FS and Mini-ImageNet, respectively. This suggests that our approach makes the least compromise on the clean data performance to attain adversarial robustness.

Dear Reviewer,

This is a gentle reminder that we have only 3 days remaining for discussions. Therefore, we kindly request you review our responses and the revisions. We have tackled all of the concerns you raised and have included additional experimental results in our previous responses.

For your convenience, we provide a short summary of our previous responses below. Thanks again for your time and effort in reviewing our work.

Best,
Author

---

Summary

W1-1. Comparison with generic adversarial training methods (i.e., TRADES) is needed.

• We additionally compared TRADES in few-shot meta-learning tasks.
• However, generic supervised training can not be easily generalized to few-shot meta-learning tasks, showing 0% unseen domain few-shot robustness.

W1-2. Other adversarial meta-training approaches with TRADES are needed.

• We additionally provide experimental results that combine TRADES in the previous adversarial meta-learning approach.
• Still, our method achieves the best performance on unseen few-shot robustness with 28.20%.

W2. How to use two encoders in test time?

• During the meta-optimization phase, we update only a single encoder. Consequently, a single encoder is employed at test time.

W3. What is FSL classifier?

• We added experimental details in the revision. We use MetaSGD FSL.

W4. Experiment comparisons are not fair.

• We provide additional experiments for other adversarial meta learning (AML) methods in the revision.
• Our method consistently shows outperforming robust transferability in non-RGB datasets, achieving an improvement of 12.65% compared to the best-forming AML baselines.

W5. Comparison with recent work [1] is needed.

• We additionally compared recent work [1] with ours.
• The evaluation setting of [1] significantly differs from ours, where [1] evaluated against much weaker attacks, FGSM with a smaller size of epsilon.
• When we use the same evaluation setting with [1], ours achieve improvements of 14.99% and 3.51% for clean and robust accuracy, respectively.

W6. Evaluation against diverse attacks is needed.

• We provide additional experiments with evaluation against diverse attacks, such as FGSM, CW, and SPSA.
• Our approach shows an outperforming average unseen robustness of 42.66% while the best-performing AML baseline (AQ) only shows an average unseen robustness of 29.23% against diverse attacks.

W7. Variance for experiments should be provided.

• We conduct multiple runs on our approach.
• The variance of converged adversarially trained models is not large, where variation is smaller than 1.0.

[1] Defensive Few-shot Learning. TPAMI 2022

Thank you for your positive comments on our contributions, which highlight clear motivation and address the significant problem with well-structed experiments.

---

In the following response, we have done our best to resolve all the concerns that you raised regarding our work. Please find our detailed response below, and if there are further concerns about our work, do not hesitate to share your comments. We would be delighted to address any additional questions or concerns you may have.

---

Weakness 1: A discussion or comparison with existing task-augmentation meta-learning methods is needed in a related section.

• Thank you for your suggestion. We have revised the related section to include suggested papers as shown in the paper. (page 3, Section 2: orange)
• For convenience, we briefly summarize the discussion in the following response.
• To achieve better generalization in tasks, previous works have proposed augmenting the tasks using rotation [1], task interpolation [2], and adversarial task up-sampling [3]. In a similar sense, our approach also implicitly brings the effect of task augmentation, enlarging the task distribution by introducing bootstrapped encoders, which naturally augment the tasks with the parameters of the encoders and could lead to generalized performance in unseen tasks.

Weakness 2: A larger number of evaluated tasks during the meta-testing.

• Thank you for your suggestion. We tested our framework under a larger number of evaluation tasks with 1000 tasks, and 5000 tasks.
• The table below demonstrates that our framework consistently delivers outperforming robust results with an increasing number of evaluation tasks compared to the best AML baseline (AQ), highlighting our approach generalizability to various types of unseen tasks.

Weakness 3: Results against different attacks, apart from PGD-20.

• Thank you for the suggestions. We will add these results to our manuscript.
• We additionally evaluated our models under CW [1], FGSM [2], and SPSA [3] as shown in the following Table. Our approach consistently demonstrates outperforming robustness against diverse attacks compared to the best-performing AML baseline (AQ).

[1] Towards Evaluating the Robustness of Neural Networks
[2] Explaining and Harnessing Adversarial Examples
[3] Adversarial Risk and the Dangers of Evaluating Against Weak Attacks

Dear Reviewer,

This is a gentle reminder that we have only 3 days remaining for discussions. Therefore, we kindly request you review our responses and the revisions. We have tackled all of the concerns you raised and have included additional experimental results in our previous responses.

For your convenience, we provide a short summary of our previous responses below. Thanks again for your time and effort in reviewing our work.

Best,
Author

---

Summary

W1. A discussion with existing task-augmentation meta-learning methods is needed.

• We have included this discussion in the main manuscript's Section on related works.

W2. A larger number of evaluation tasks should be considered during meta-testing.

• We tested on a larger number of evaluation tasks (400, 1000, and 5000 tasks) and obtained consistent results.

W3. Robust evaluation against diverse attacks is needed

• We further evaluated our models under CW, FGSM, and SPSA.
• Our approach shows an outperforming average unseen robustness of 42.66% while the best-performing AML baseline (AQ) only shows an average unseen robustness of 29.23% against diverse attacks.

W1-1. The proposed method has little contribution other than introducing already known technical components into the meta-learning problem setting.

• There seems to be a misunderstanding of our approach. The concept of each component of ours and the [a,b] is even a totally different concept.
• First of all, our bootstrapped encoders are augmented encoders that diversify the task distribution and are different from the [a] ensemble models which are several individual classifiers. There are none of the single similar concepts with [a].
• Furthermore, our multi-view latent attack is a latent adversarial attack that maximizes the discrepancy in the feature level while training the model in the supervised meta-learning approach. In contrast, [b] is a contrastive learning approach that tackles obtaining robustness without class information by introducing and maximizing the similarity of the individual instance latent.
• This approach could seemingly be similar to ours but as we clearly demonstrate as shown in Section 3.3 and Table 3, [b] could not leverage generalized robustness in meta-learning tasks because very few-shot instances are the bottleneck for contrastive learning that requires large batch size.
• In particular, our approach is not simply introducing these technical components into the meta-learning approach as we clearly stated in Section 3.3, and also provides empirical results in Table 3. Therefore, it is quite unfair that our approach is simple technical combinations while we provide extensive experimental ablation and clear descriptions. We politely ask a reviewer to find the innovative ideas that we proposed to solve the generalized adversarial robustness in few-shot tasks.

W1-2. The reason why meta-learning is needed is unclear. Comparison with a baseline pre-trained with adversarial training should be provided. To the best of our knowledge, none of the adversarial self-supervised learning has shown generalizability on few-shot tasks. We tackle the problem of generalized adversarial robustness in the scope of the meta-learning approach.

Q1. Is meta-training really necessary?

• We understand your perspective that an adversarial self-supervised learning approach could be another research direction to tackle generalized robustness. However, this does not justify that meta-training is not a necessary research direction, especially since none of the research has solved the generalized robustness problem.
• Furthermore, meta-training costs (18 hours with a single 2080 Ti GPU) way less than adversarial self-supervised learning (Takes 36 hours or more with two 2080 Ti GPUs). The required data for meta-training (only trained on 64 classes of CIFAR-100) is less than adversarial self-supervised learning (trained on 100 classes of CIFAR-100). Meta-learning is more applicable in the real world when the data is expensive or scarce such as clinical problems.
• From the experimental results, we believe we have shown the significance and necessity of meta-training on generalized adversarial robustness tasks. |Method|CIFAR-FS|(Seen)|Mini-ImageNet||Tiered-ImageNet||CUB||Flower||Cars||Unseen Avg.|| |-|-|-|-|-|-|-|-|-|-|-|-|-|-|-| ||Clean|Rob.|Clean|Rob.|Clean|Rob.|Clean|Rob.|Clean|Rob.|Clean|Rob.|Clean|Rob.| |TRADES|44.21|22.81|23.68|10.63|23.22|13.43|27.05|7.53|36.98|18.21|25.74|8.63|27.33|11.69| |RoCL [b]|70.61|44.48|44.20|19.13|46.59|23.36|48.99|20.89|78.43|50.21|46.63|20.72|52.97|26.86| |Ours|67.75|43.42|45.82|24.12|51.46|30.06|48.56|25.23|66.49|42.16|38.29|19.43|50.32|28.20|

W2, Q2. One more encoder for multi-view representation learning, increasing the capacity of models where comparison is not fair.

• There seems to be a misunderstanding regarding how our bootstrapped encoders are employed during the training.
• Our approach uses the same parameter capacity. As noted in the algorithm, our encoders operate as bootstrapped parameters during the inner-adaptation which are shared parameters. (page 5: algorithm last line/ page 5: green text)
• Consequently, our approach has the same capacity as previous works with the same backbone ResNet12. (page 7: orange text) During the meta-optimization phase, we update only a single encoder. (page 6: green text)