Adapting Multi-modal Large Language Model to Concept Drift From Pre-training Onwards

We sincerely appreciate your insightful and comprehensive review. Your positive feedback serves as a significant encouragement for our team, acknowledging 1) the clear writing, 2) extensive experiments and 3) substantial improvements in experiment results. Additionally, we have meticulously addressed each of your questions in detail, incorporating your feedback into our manuscript revisions.

1. Downstream tasks based on VL models. Thanks for your sincere review. We apologize for the misunderstanding of the experiments. We have provided results of some methods based on VL models in Table.1, such as LPT [1], BALLAD [2], Decoder [3], VL-LTR [4] and LiFT [5]. We have revised our paper to clarify the misunderstanding in LINE 371 as shown below. Besides, we also provided more results of CLIP under zero-shot, linear probing and fine-tuning to compare our method.

   LINE 371. It is worth noting that, most vision language models only focus on the impact of tailed drift in the fine-tuning, such as LPT [1], BALLAD [2], Decoder [3], VL-LTR [4] and LiFT [5]. We are the pioneers in revealing the unexplored impacts of concept drift from pre-training onwards.

2. Ablations studies of the T-distributed spherical adapter. Thank you very much for your constructive suggestions. In the ablation experiment, we investigated the efficacy of the T-distributed adapter in addressing the concept drift challenge in vision language models from two perspectives: by examining the impact of removing this module in pre-training and downstream tasks in section 3.3.1, and by conducting an ablation study on the internal concentration parameter kappa of the T-distributed adapter in section 3.3.2. The results verify that the T-distribution adapter's impact is more pronounced in the fine-tuning stage compared to pre-training, since the model is directly involved in specific downstream tasks, and explicit category centers are present in the classifier. In contrast, pre-training primarily focuses on aligning image-text features, where the implicit information of categories is embedded. Additionally, we find $\kappa=16$ yields superior results. We argue that a smaller concentration makes it challenging to effectively mitigate the biases introduced by concept drift in the vision language model. We have revised our paper to further explain the ablation experiments in LINE 512.

3. Domain Shift Results. Sincerely thanks for your thoughtful and constructive suggestions. Your feedback improves us a lot. We add experiments on ImageNet-Sketch to evaluate the performance of our method on domain shift in LINE 452.

   CLIP+ZS [6]	CLIP+LP [6]	CoOp [7]	VPT [8]	DAPT [9]	Ours
   46.1	36.0	47.1	47.7	48.3	50.2

   Moreover, we evaluate the generalizability of our method in the domain generalization setting. Experiments are conducted on ImageNet-Sketch [10] with ImageNet [11] as the source dataset, as shown in the Table above. From the experiment results, our method achieves superior performance with an accuracy of 50.2% on ImageNet-Sketch, attributed to the robustness of the T-distribution-based drift adapter. It further verifies the generalization ability of our model in the open-world.

We appreciate your time and thorough review. Your feedback is highly valuable to us, and we welcome further communication from you. If you are satisfied with our response, we would be grateful for your support in enhancing our rating score.

[1] Dong, Bowen, et al. "LPT: Long-tailed prompt tuning for image classification." ICLR 2023.

[2] T. Ma et al., “A Simple Long-Tailed Recognition Baseline via Vision-Language Model,” Nov. 29, 2021, arXiv: arXiv:2111.14745. doi: 10.48550/arXiv.2111.14745.

[3] Y. Wang et al., “Exploring Vision-Language Models for Imbalanced Learning,” IJCV 2024.

[4] C. Tian, W. Wang, X. Zhu, J. Dai, and Y. Qiao, “VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognition,”ECCV 2022.

[5] J.-X. Shi, T. Wei, Z. Zhou, J.-J. Shao, X.-Y. Han, and Y.-F. Li, “Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts,” ICML 2024.

[6] A. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML 2021.

[7] K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Learning to Prompt for Vision-Language Models,” IJCV 2022.

[8] M. Jia et al., “Visual Prompt Tuning,” ECCV 2022.

[9] E. Cho, J. Kim, and H. J. Kim, “Distribution-Aware Prompt Tuning for Vision-Language Models,” ICCV, 2023.

[10] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. "Learning robust global representations by penalizing local predictive power." NeurIPS, 2019.

[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. "Imagenet: A large-scale hierarchical image database." CVPR, 2009.

I've read the authors' response and have increased my score given my concerns have been largely addressed.

Sincerely thanks for your meaningful and detailed review. Your positive feedback is a huge encouragement to our team, including 1) clear issue presented, 2) motivated method and 3) effective validation. And, we have provided detailed responses below to address your questions one by one, and made revisions to our manuscript according to your feedback, highlighting these changes in red for your reference in the revised version.

1. Comparison with CLIP. Thank you very much for your valuable suggestions. We have revised our Table 1 to add results of CLIP on ImageNet-LT and iNatualist2018 as shown below, and we have added the discussion about the comparison between our method and CLIP in Section 3.1 as shown below. Besides, we have added a detailed introduction to CLIP in related works.

   ImageNet-LT	Many	Medium	Few	All
   CLIP + Zero-Shot	82.0	70.4	69.6	70.5
   CLIP + Linear Probing	87.3	65.1	19.0	67.4
   CLIP + Fine Tuning	83.0	65.0	39.9	68.5

   iNatualist2018	Many	Medium	Few	All
   CLIP + Zero-Shot	9.9	5.3	4.6	5.5
   CLIP + Linear Probing	62.4	7.1	0.1	10.0
   CLIP + Fine Tuning	79.4	67.6	59.1	65.4

   LINE 417: We compare our method with the CLIP [1] under zero-shot, linear probing and fine-tuning, and CLIP results are from the Decoder [2]. Based on the zero-shot results, it is evident that CLIP, even trained on large-scale and high-quality WIT datasets, struggles to address the issue of tailed drift. CLIP only achieves 5.5% in iNaturalist2018, and the accuracy variance between many-shot and medium-shot scenarios is 11.6% in ImageNet-LT. Our method significantly outperforms CLIP in dealing with tailed drift, especially in iNaturalist2018. It also indicates that training on a high-quality balanced dataset alone cannot effectively mitigate the bias induced by tailed drift. Furthermore, the results of linear probing and fine-tuning demonstrate that imbalanced datasets can induce pronounced tail drift in MLLMs that damage the model performance. The CLIP accuracy in Few-shot is only 39.9% under fine-tuning on ImageNet-LT, much lower than the 69.6% under zero-shot. It further verifies the challenges brought by imbalanced data in the training of MLLMs and the superiority of our method in adapting the MLLM to concept drift from pre-training onwards.

   LINE 935: Related Work. CLIP [1] was introduced to separately extract features from the visual encoder and the text encoder, and combine them using contrastive learning. CLIP supports a variety of downstream tasks, including image retrieval, image classification tasks and especially zero-shot classification tasks. But, it cannot generate detailed captions based on images due to the lack of a text decoder. In contrast, our model primarily addresses the concept drift issue within multi-modal large language models, since an image-grounded text decoder is employed to generate text based on the images. Besides, CLIP requires a large-scale and high-quality WIT dataset to be driven, that contains 37.6 million entity image-text samples with 11.5 million unique images across 108 Wikipedia languages. Whereas, our method is validated under the extended ImageNet-LT, which consists of only 115.8K imbalanced images-text pairs.

We sincerely appreciate your thoughtful and detailed review. Your positive feedback serves as a great source of encouragement for our team, especially recognizing the 1) novel contribution to concept drift in MLLMs, 2) effective mitigation strategies, 3) extensive experiments and 4) new datasets for long-tailed open-world settings. Furthermore, we have diligently responded to each of your questions and incorporated your feedback into our manuscript revisions, with these changes highlighted in red for your reference in the revised version.

• Relevance of Long-Tail and OOD Issues in Pre-Training. Thanks for your kind suggestions. As shown in Figure 1 of our manuscript, we conduct a comparison experiment of the VL model trained on the balanced dataset ImageNet and the imbalanced dataset ImageNet-LT, to illustrate the relevance of long-taile and OOD in the pre-training. It is evident that training on the imbalanced dataset leads to a higher degree, indicating worse intra-class compactness brought by the tailed drift. It leads to an overall performance degradation in the pre-training. Beyond that, the undistinguished OOD drift will bias the underlying distribution of the feature space in the VL model, further disturbing the image-text alignment in the pre-training.

• Sub-Distribution in Training Data. Many thanks for your kind review. We indeed take into account the different sub-distributions present in the training data. It is mainly reflected in the robustness and generalization ability of our model. We have revised our paper and added experiments on ImageNet-Sketch to evaluate the performance of our method on robustness and generalization as shown below. It verifies the robustness and generalization of our model in dealing with sub-distribution in training data.

  CLIP+ZS [6]	CLIP+LP [6]	CoOp [7]	VPT [8]	DAPT [9]	Ours
  46.1	36.0	47.1	47.7	48.3	50.2

• Impact of Long-Tail Issues with Large Datasets. Thanks for your constructive suggestions. When the training data is large, the large model still cannot avoid the adverse effects of long-tail data on the model. We have added results of CLIP under zero-shot to explain the question. Even though CLIP is training in large-scale and high-quality WIT dataset, it only achieves 5.5% accuracy in iNatualist2018 dataset, further demonstrating that MLLMs still suffer from the tailed drift, despite the training datasets being large.

• Related Work. Thanks for your kind advice. We have revised our paper, discussed the related work of H-SRDC [1] and cited this paper in LINE 1028. H-SRDC [1] enhances intra-class compactness by combining target data clustering with a domain-shared classifier and cluster centroid learning, enhancing deep clustering by minimizing Kullback-Leibler divergence between network predictions and an auxiliary distribution. In contrast, We utilize the light-tailed characteristics of the T-distributed adapter, which effectively counteracts the squeezing of tail categories caused by an overwhelming number of head samples, thereby alleviating the bias induced by tailed concept drift.

• Pre-training in Figure 1 (a). Thanks for your detailed suggestions. We have revised our paper and added more detailed descriptions about the test sets and OOD samples in the caption of Figure 1. Firstly, we use the same test sets to evaluate the balanced and long-tailed trained VL models to illustrate the impact of tailed drift ti MLLMs. And these test sets are balanced in terms of class distribution. Second, OOD samples are not included in the pre-training of the VL model. We argue that VL models will suffer from OOD drift during inference, so we use OOD data during evaluation. And we find that compared to training on the balanced dataset, the VL model trained under an imbalanced scenario is harder to distinguish between ID samples and OOD samples from the open world due to their similar inter-class separability.

• Fine-tuning in Figure 1 (b). Many thanks for your advice. We have revised our paper and added more detailed descriptions about the test sets and OOD samples in the caption of Figure 1.. The VL model used for evaluation is fine-tuned on imbalanced datasets.

• Proofreading. Thank you very much for your feedback. We have meticulously proofread the entire paper to enhance its clarity and coherence, involving rectifying any typos and conducting a thorough review of the manuscript to ensure it is error-free.

Thank you for your continued time and thorough review. Your feedback is immensely valuable to us, and we are always eager to hear from you. If you find our response satisfactory, could you please consider helping us improve our rating score?

[1] Towards Uncovering the Intrinsic Data Structures for Unsupervised Domain Adaptation using Structurally Regularized Deep Clustering. TPAMI, 2022.

I have reviewed the authors' rebuttal as well as the comments from the other reviewers. I appreciate the thorough discussions, analyses, and experiments presented, which effectively address many of my concerns. The revised paper does adequately incorporate the suggested works for discussion or experiments, which is good. In light of these, I will raise my current rating.

Thank you very much for your review, especially for summarizing the strengths of our work, including 1) motivation 2) detailed experiments and 3) good performance. In response to your concerns, we have provided a detailed explanation below and revised our manuscript accordingly, with all changes highlighted in red in the revised version.

1. Weaknesses 1: image-grounded decoder and text modeling loss. Thanks for your kind suggestions. The motivation of the image-grounded decoder is applying text modeling to train the vision language model driven by the text modeling loss, in addition to aligning image-text by contrastive learning. They are introduced in Blip [1] to implement both vision-language understanding and generation tasks, and we are following them to generate text based on images. We added the following descriptions in the Section 2.3 of the revised manuscript to clarify the misunderstanding, marked in red. Thanks for your advice again.

   LINE 292: Additionally, an image-grounded text decoder is employed to produce a textual description corresponding to a provided image. Utilizing the input visual features $x_{img}$ and text features $x_{txt}$, we initially create fused multi-modal representations by merging the image and text feature embeddings. These combined features act as the keys and values within the cross-attention blocks in the image-grounded text decoder. Through conditioning on the already predicted partial sequence $y_{i<j}$, the decoder iteratively forecasts the token at position $j$, effectively producing textual descriptions corresponding across modalities.

   LINE 307: Additionally, coupled with the T-distributed adapter, language modeling loss is utilized to activate the image-grounding text encoder for generating coherent and detailed captions based on the image, further propelling the training of all three modules. Driven by language modeling loss, the model is trained to optimize a cross-entropy loss with label smoothing, to maximize the likelihood of the generated text in an autoregressive manner.

2. Weaknesses 2: The setup of the experiment. Thanks very much for your detailed and thoughtful review. We found it is similar to the Question 1, so we answered both questions together here. We will answer your questions from the following three aspects:

   • Additional data. We sincerely apologize for the misunderstanding caused to you. Despite our created OpenMMlo extends ImageNet-LT, iNatualist2018 and Places-LT with elaborate descriptions, we did not combine three datasets to pre-train the model. When we conduct long-tail evaluation, we still use the same dataset as other methods without additional data. We have revised our manuscript to clarify the misunderstanding in LINE 358 of Section 3.1, marked in red.

   • Larger resolution. Thanks for your kind suggestions. Your feedback is essential to our improvement process. We have revised Table 1 in our revised manuscript, which added the results with the resolution of $224\times 224$ as shown below. Even at low resolution, our method still outperforms other methods when training from scratch or only fine-tuning. Besides, we follow the LiVT [3] to present the results with a large resolution, showing our model can also benefit from a larger image input.

     	Many	Medium	Few	All
     Training from Scratch
     ImageNet-LT	76.4	66.2	48.9	68.0
     iNaturalist 2018	82.5	79.8	77.1	78.9
     Only Fine-tuning
     ImageNet-LT	79.5	76.5	74.1	77.2
     iNaturalist 2018	83.5	82.2	80.7	81.7

   • Additional modules. Thanks very much for your suggestions, and we fully understand your concerns. However, the additional module of the T-distributed adapter does not have learnable parameters, that we only tested kappa as a learnable parameter in the ablation experiment. Therefore, the improvement of our methods is not attributed to the additional parameters from extra modules.