MetaCoCo: A New Few-Shot Classification Benchmark with Spurious Correlation

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point and try our best to update the manuscript accordingly.

[W1] Contributions are not clearly and accurately stated.

Response: We thank the reviewer for pointing out this issue. We have added a concise and clear contribution summary of our paper in the “Introduction” section. In a nutshell, our main contributions can be summarized as follows:

• We propose a large-scale few-shot classification benchmark with spurious-correlation shifts arising from various contexts in real-world scenarios, named MetaCoCo, in which each class (or concept) consists of different backgrounds (or contexts).

• We further propose a metric by using a pre-trained vision-language model to quantify and compare the extent of spurious correlations on MetaCoCo and other benchmarks.

• We conduct extensive experiments on MetaCoCo to evaluate and compare most recent methods in few-shot classification, domain shifts, overfitting, and self-supervised learning.

[W2] How to understand the conceptual and contextual information mentioned in the manuscript?

Response: Thank you for the comment. In this paper, we follow the existing literature addressing out-of-distribution problems in computer vision [1, 2, 3] to adopt the terminologies “conceptual information” and “contextual information”. Specifically, the conceptual information in an image refers to the object of the class, while the contextual information in an image refers to the background or the environment. For example, consider an image with a dog in green grass, and the true class label is "dog". Then "dog's nose" can be regarded as a conceptual information, while "green grass" can be regarded as a contextual information. As shown in Figure 1 (b), the conceptual information includes “dog”, “bicycle”, “train”, “cat”, etc., whereas the contextual information includes “autumn”, “snow”, “rock”, “water”, etc.

[W3] The explanation for the spurious-correlation few-shot classification is not easy to understand.

Response: We continue with our response to W2 to further explain the spurious-correlation in few-shot classification. For the aforementioned example, that is, an image with a dog in green grass, a well-trained classifier may capture both conceptual and contextual information: "dog's nose" and "green grass". However, given that the true label of the image is “dog”, we know that the classifier produces a spurious correlation between the contextual information "green grass" and the image label “dog”.

Different from the widely known Meta-dataset with cross-domain shifts shown in Figure 1 (a), the proposed MetaCoCo as shown in Figure 1 (b) further emphasizes the existence of spurious-correlations in the real-world scenarios. For example, in base classes we have an image "dog in green grass" (and the true class label is "dog"), while in novel classes we instead have an image "cat in green grass" (and the true class label is "cat"). Thus, we need the classifier to be able to distinguish between these conceptual and contextual information.

[W5] The network structure of ResNet-50 is not explained in EXPERIMENTAL SETUP, it is suggested that the author explain it in backbone architectures.

Response: We thank the useful suggestions from the reviewer. For network structure, we follow [18] to implement ResNet50 using 50 layers, including 48 convolutional layers and 2 fully connected layers, with the detailed network structure summarized in below.

Specifically, each layer in conv2_x and conv5_x are repeated with 3 times, each layer in conv3_x are repeated with 4 times, and each layer in conv4_x are repeated with 6 times. The building block of ResNet50 is the residual block, which typically contains two convolutional layers along with the skip connection. The network architecture includes global average pooling and a softmax layer for classification in the final layers.

For all experiments, we use the A100 80G GPU as the computational resource. We tune batch size in $\{128, 256, 512\}$. For learning rate, optimizer, and weight decay, we follow [20,21] as a comprehensive library for few-shot learning to implement using fixed parameter combinations as summarized in below.

For ease of reproducibility, we also added Table 4 in Appendix C showing the optimal parameter combinations when implementing the other 17 methods using ResNet12 as the backbone model following [19].

We hope the above discussion will fully address your concerns about our work, and we would really appreciate it if you could be generous in raising your score. We look forward to your insightful and constructive responses to further help us improve the quality of our work. Thank you!

Reference

[1] Haotian Wang et al. Out-of-distribution generalization with causal feature separation. TKDE 2023.

[2] Xingxuan Zhang et al. Deep Stable Learning for Out Of Distribution Generalization. CVPR 2021.

[3] Mingjun Xu et al. Multi-view Adversarial Discriminator: Mine the Non-causal Factors for Object Detection in Unseen Domains. CVPR 2023.

[4] Tianjun Ke et al. Revisiting Logistic-softmax Likelihood in Bayesian Meta-learning for Few-shot Classification. NeurIPS 2023.

[5] Jiangtao Xie et al. Joint Distribution Matters: Deep Brownian Distance Covariance for Few-Shot Classification. CVPR 2022.

[6] Ji Zhang et al. DETA: Denoised Task Adaptation for Few-Shot Learning. ICCV 2023.

[7] Long Tian et al. Prototypes-oriented Transductive Few-shot Learning with Conditional Transport. ICCV 2023.

[8] Wertheimer Davis et al. Few-shot classification with feature map reconstruction networks. CVPR 2021.

[9] Zhanyuan Yang et al. Few-shot classification with contrastive learning. ECCV 2022.

[10] Heng Wang et al. Revisit Finetuning strategy for Few-Shot Learning to Transfer the Emdeddings. ICLR 2023.

[11] Radford Alec et al. Learning transferable visual models from natural language supervision. ICML 2021.

[12] Kaiyang Zhou et al. Learning to prompt for vision-language models. IJCV 2022.

[13] Kaiyang Zhou et al. Conditional prompt learning for vision-language models. CVPR 2022.

[14] Haotao Yao et al. Visual-Language Prompt Tuning with Knowledge-guided Context Optimization. CVPR 2023.

[15] Yassine Ouali et al. Black Box Few-Shot Adaptation for Vision-Language models. ICCV 2023.

[16] Beier Zhu et al. Prompt-aligned Gradient for Prompt Tuning. ICCV 2023.

[17] Muhammad Uzair Khattak et al. MaPLe: Multi-modal Prompt Learning. CVPR 2023.

[18] Kaiming He et al. Deep Residual Learning for Image Recognition. CVPR 2016.

[19] Wenbin Li et al. Libfewshot: A comprehensive library for few-shot learning. TPAMI 2023.

[20] Kaiming He et al. Momentum contrast for unsupervised visual representation learning. CVPR 2020.

[21] Ting Chen et al. A simple framework for contrastive learning of visual representation. ICML 2020.

[W4] How recent methods perform on the MetaCoCo dataset?

Response: We thank the useful suggestions from the reviewer. We added more experiments to compare the performance of the 8 most recent few-shot classification methods on the proposed MetaCoCo. The experimental results are shown as below.

From the table, we find that: (1) The performance of these methods is significantly reduced on our proposed MetaCoCo dataset. (2) The performance of fine-tuning based methods is better than other methods, such as LP-FT-FB [10].

In addition to the methods mentioned by the reviewer, we also verify diverse methods including vision-language pre-trained methods.

The above table shows the experimental results of VLMs on MetaCoCo, where base and new represent the accuracy of the base and new classes, respectively. HM is the harmonic mean of accuracy on base and new data. From the above table, we find two interesting phenomenons: (1) The more samples are used in the model prompt-tuning phase, the worse the performance of the model will be. For example, the performance drops from 62.40% with 1-shot to 43.98% with 16-shot in CoOp with ResNet50. This further indicates that the more samples used, the greater the shifts in spurious correlations introduced, resulting in reduced model performance. (2) The performance of prompt-tuning methods is lower than zero-shot CLIP in all settings. The possible reason is that prompt-tuning methods overfit to spurious information, therefore, affecting the generalization ability of the model.

The replies are very detailed and mostly addressed my concerns. I've raised my score to 8.

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point and try our best to update the manuscript accordingly.

[W1] The technical contribution of this work would be more significant if a model can be provided to address the problem.

Response: We thank the reviewer for pointing out this issue. Nonetheless, we kindly remind our paper is submitted to the “datasets and benchmarks” track in ICLR 24, and we agree with the reviewer that the main contribution is the proposed large-scale few-shot classification benchmark collected from real-world scenarios, rather than a specific method. To the best of our knowledge, this is the first benchmark for studying the spurious-correlation shifts in few-shot classification. Moreover, we further propose a novel metric by using a pre-trained vision-language model to measure the extent of spurious correlations.

To make our contributions more recognizable, we have added a concise and clear contribution summary of our paper in the “Introduction” section. In a nutshell, our main contributions can be summarized as follows:

• We propose a large-scale few-shot classification benchmark with spurious-correlation shifts arising from various contexts in real-world scenarios, named MetaCoCo, in which each class (or concept) consists of different backgrounds (or contexts).

• We further propose a metric by using a pre-trained vision-language model to quantify and compare the extent of spurious correlations on MetaCoCo and other benchmarks.

• We conduct extensive experiments on MetaCoCo to evaluate and compare most recent methods in few-shot classification, domain shifts, overfitting, and self-supervised learning.

[W2] What is the difference between spurious-correlation studied in this paper and overfitting in few shot learning?

Response: Thank you for raising such an interesting and insightful concern. In our opinion, we consider overfitting to be a broader concept than spurious-correlation, that is, spurious-correlation in the training data can cause model overfitting when naively using empirical risk minimization, but not necessary. In fact, many other factors can also lead to model overfitting, such as long-tailed data [1], noisy labels [2], cross-domain shifts [3, 4], etc.

[W3] Can we view spurious-correlation as one type of model overfitting?

Response: Yes, please kindly refer to our response to W2 that the spurious-correlation problem is one of the causes resulting in model overfitting.

[W4] If this is true, it would be interesting to have a comprehensive study of overfitting problem in few-shot learning, in addition to the spurious correlation studied in this paper.

Response: As suggested by the reviewer, we added a comprehensive study comparing 9 most recent methods for addressing the overfitting problem in few-shot learning on MetaCoCo. The experimental results are shown as below.

From the table, we find that these methods do not show excellent performance on MetaCoCo with spurious-correlation. One possible reason is that these methods aim to address the overfitting problem of base classes in the few-shot training phase, but not for the spurious-correlation problem, making the trained model tend to overfit to the context information, e.g., background knowledge.

We hope the above discussion will fully address your concerns about our work, and we would really appreciate it if you could be generous in raising your score. We look forward to your insightful and constructive responses to further help us improve the quality of our work. Thank you!

We thank the reviewer for the detailed and insightful comments. We kindly remind you that the author-reviewer discussion period is about to close and we hope that the response in the rebuttal could address all your concerns. Specifically, we added a concise and clear contribution summary of our paper (in Introduction, Section 1) and a comprehensive study comparing the 9 most recent methods for addressing the overfitting problem in few-shot learning on MetaCoCo (in Appendix D) to better address your concerns. We also added extensive experiments comparing the 9 most recent few-shot classification methods on MetaCoCo (in Table 2, Section 5), as well as the 7 most recent vision-language pre-trained models (VLMs) using ResNet50 and ViT-B/16 as the vision backbone, and further clarified our experimental details in terms of the backbone architectures and parameter tunings (in Appendix C). We hope the above discussion will fully address your concerns about our work.

We are anticipating any further advice, inquiries or remaining concerns you may have. We will make our best effort to handle them. Thank you!

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point and try our best to update the manuscript accordingly.

[W1] The performance of pre-trained large models like CLIP on the proposed MetaCoCo dataset.

Response: We thank the reviewer for pointing out this issue. We added extensive experiments for evaluating the performance of vision-language pre-trained models (VLMs) on our proposed MetaCoCo dataset. These VLMs are broadly categorized into zero-shot CLIP and prompt-tuning CLIP. We follow the original papers to implement using ResNet50 and ViT-B/16 as backbones with 1-shot and 16-shot samples, and the experimental results are shown as below.

In the above table, “base” and “new” represent the accuracy of the base and new classes, respectively, and HM is the harmonic mean of accuracy on “base” and “new” data. From the table, we find two interesting phenomenons: (1) The more samples are used in the model prompt-tuning phase, the worse the performance of the model will be. For example, the performance drops from 62.40% with 1-shot to 43.98% with 16-shot in CoOp with ResNet50. This further indicates that the more samples used, the greater the shifts in spurious correlations introduced, resulting in reduced model performance. (2) The performance of prompt-tuning methods is lower than zero-shot CLIP in all settings. The possible reason is that prompt-tuning methods overfit to spurious information, therefore, affecting the generalization ability of the model.

We hope the above discussion will fully address your concerns about our work. We look forward to your insightful and constructive responses to further help us improve the quality of our work. Thank you!

References

[1] Radford Alec et al. Learning transferable visual models from natural language supervision. ICML 2021.

[2] Kaiyang Zhou et al. Learning to prompt for vision-language models. IJCV 2022.

[3] Kaiyang Zhou et al. Conditional prompt learning for vision-language models. CVPR 2022.

[4] Haotao Yao et al. Visual-Language Prompt Tuning with Knowledge-guided Context Optimization. CVPR 2023.

[5] Yassine Ouali et al. Black Box Few-Shot Adaptation for Vision-Language models. ICCV 2023.

[6] Beier Zhu et al. Prompt-aligned Gradient for Prompt Tuning. ICCV 2023.

[7] Muhammad Uzair Khattak et al. MaPLe: Multi-modal Prompt Learning. CVPR 2023.

We thank the reviewer for the detailed and insightful comments. We kindly remind you that the author-reviewer discussion period is about to close and we hope that the response in the rebuttal could address all your concerns. Specifically, we added experiments comparing 7 most recent vision-language pre-trained models (VLMs) using ResNet50 and ViT-B/16 as the vision backbone, respectively. We hope the above discussion will fully address your concerns about our work.

We are anticipating any further advice, inquiries or remaining concerns you may have. We will make our best effort to handle them. Thank you!

We are glad to know that your concerns have been effectively addressed. We are very grateful for your constructive comments and questions, which helped improve the clarity and quality of our paper. Thanks again!