Bridge the Modality and Capability Gaps in Vision-Language Model Selection

We appreciate the reviewer's comments. Below are our responses:

• Q1：Use open-source datasets may violate zero-shot data restrictions.

• A1:

  1. Firstly, we want to emphasize that our method focuses on how to use open-source datasets to help us select the appropriate models from the VLM Zoo for the current task, rather than using open-source datasets to enhance the zero-shot image classification capabilities of pre-trained VLMs. For the pre-trained VLMs we selected, we directly complete the image classification tasks on the target datasets without using any additional data or undergoing any extra training process. Therefore, we do not introduce additional data when the VLM performs image classification tasks, and there is no issue with data leakage.

  2. Secondly, the open-source datasets we use are all common and widely used. And since the pre-traning phase of VLMs have also used a lot of open-source data, it is likely that VLM's pre-training data has already included the data similar to the open-source datasets we use. Therefore, we believe that our use of common open-source datasets is reasonable. It is important to emphasize that the previous SoTA method, ModelGPT[1], used the same open-source datasets as we did, ensuring that our comparison with their results is fair. Moreover, we believe that in the era of large models, a developmental trend is that we should make the most of the data and resources that are easily accessible to us.

• Q2: Whether the performance of the proposed method is still remarkable if the categories in the open-source datasets are very different from the ones in the target zero-shot classification tasks.

• A2:

  The key insight of our method is to leverage the category relatedness to reuse the model's performance evaluation results from previous tasks. Therefore, for a completely new dataset that has no correlation with the open-source datasets, we might not expect it to achieve outstanding performance, just as different algorithms always have their suitable application scenarios. However, we think this issue will not cause a drastic deterioration in our method's performance for the following reasons:

  • For most tasks, it is generally possible to find tasks that have a certain degree of correlation with them. This correlation can be at the object category level (e.g., tigers and leopards are both large felines) or at the attribute level (e.g., we found that in our dataset, the accuracy of VLM in recognizing the "striped" pattern in the DTD [2] dataset is positively correlated with its accuracy in recognizing "tiger" and "zebra" in Imagenet [3]. This is because the skin patterns of zebras and tigers are striped).

  • For some new datasets that indeed have no correlation with open-source datasets, we can first measure their degree of correlation with the open-source datasets by calculating the text similarity between their class names and those of the open-source datasets. Then, we can set a threshold. Then for open-source tasks with a correlation score lower than the threshold, we do not use our SWAB method, and the method degrades to the original SoTA ModelGPT [1] method. For open-source tasks with a correlation score higher than the threshold, we use our SWAB method to leverage the open-source datasets to help us better select the model.

  • As VLMs are evaluated on an increasing number of tasks, their performance evaluation results on previous tasks are expanded, further enhancing the capability of our method. This creates a virtuous cycle.

[1] Zohar, et al. "Lovm: Language-only vision model selection." NeurIPS 2023.

[2] Cimpoi, Mircea, et al. "Describing textures in the wild." CVPR 2014.

[3] Deng, Jia, et al. "Imagenet: A large-scale hierarchical image database." CVPR 2009.

Dear Reviewer mpun,

We hope this message finds you well. I am reaching out to kindly request your prompt response to confirm whether our responses adequately address your queries. We sincerely thank you for your time and effort during this discussion period. Your timely feedback is greatly appreciated.

Dear reviewer mpun,

We thank you sincerely for your time and effort in reviewing our manuscript and providing valuable suggestions. As the author's response period is nearing its conclusion, we kindly remind you to review our response.

We have provided detailed responses to your questions and hope that they adequately address your concerns. If you need further clarification or have any other questions, please feel free to discuss them with us! We are more than willing to continue our communication with you. We would greatly appreciate it if you would update the rating by taking our responses into account.

Dear Reviewer mpun,

We have responded to your comments in detail. As the discussion period will end in less than one day, we would like to ask whether there are any additional concerns or questions we could address.

Thanks very much for your effort!

Best regards,

Authors

We appreciate the reviewer's comments. Below are our responses:

• Q1: Why not use combination of images and texts (or even text-image pairs) for VLM selection？

• A1:

  1. Firstly, we want to emphasize that in our paper, "Language-Only" means that for the target new task's dataset, we can only obtain the text descriptions (e.g. category names) of the target task. And we can't obtain the images of the target task. This has been explained in lines 29-31 of our paper. But for the open-source datasets, we did use the image-text pairs mentioned by the reviewer.

  2. Secondly, we follow the setting proposed by LOVM [1] in NeurIPS 2023, rather than the setting we devised ourselves. This setting has already been peer-reviewed and validated.

  3. Thirdly, benefiting from image-text pre-training on large-scale data, VLMs represented by CLIP have strong zero-shot capabilities. Therefore, users naturally expect to obtain a model with strong zero-shot capabilities without providing image data. And this aligns with our "Language-Only" setting. This highlights the rationality and practical significance of the setting in our paper.

  4. Finally, we want to emphasize that generating diverse text with LLMs is simpler and more cost-effective than collecting precisely annotated diverse images. To obtain accurate evaluation results, the test image samples need to be sufficiently diverse and accurately labeled. However, collecting such diverse and precisely labeled image samples is quite challenging and may require significant human and financial resources. Images or image-text pairs scraped from the web can be extremely noisy, as pointed out in the CLIP[2], ALBEF[3], and BLIP[4] papers. And as mentioned in the introduction of our paper and by reviewer qHTS, collecting these images is even more difficult for some downstream users, such as non-machine learning practitioners. What they want is to input their task description into a search box, and have a model selection algorithm return the suitable VLMs for their task.

  Compared to collecting images, generating diverse, high-quality text with LLMs is relatively easy. This opinion is also mentioned in some work[5]. Our paper further analyzes the critical issue of "how to make the diverse text generated by LLMs better substitute for diverse images" and provides a solution, thereby demonstrating significant value and importance of our work.

• Q2: Why only choose the most suitable VLM?

• A2:

  1. First, we want to emphasize that our method focuses on model selection, specifically providing an accurate estimate of the performance ranking of VLMs in the VLM Zoo for a target task. This performance ranking estimate helps users make better use of the VLM Zoo for their target tasks, whether selecting the single best model or choosing the Top-K models for an ensemble. In fact, the reviewer's question is concerned with "how to use the VLM Zoo based on the obtained ranking list," while our method focuses on "how to accurately estimate the performance ranking of VLMs in the VLM Zoo for the target task." These are different points of focus.

  2. Secondly, our article prioritizes selecting a single best model mainly because the inference cost of a single model is smaller compared to an ensemble of multiple models, aligning better with the reality that most downstream users lack sufficient computational resources. However, users can still use the performance ranking list provided by our method to select the top-K models for ensemble. We reiterate that the ranking list estimates provided by our method can universally assist various uses of the VLM Zoo, including using a single model or an ensemble.

• Q3: Why only focus on the image classification task?

• A3:

  1. Firstly, as mentioned in LOVM[1], the reason we do not primarily focus on tasks such as segmentation and object detection is that these tasks usually require the introduction of additional model structures and training processes on top of the original VLM pre-trained model. This leads to additional noise when evaluating VLM performance (for example, using different model fine-tuning methods may result in completely different VLM performance rankings, making it difficult to determine the true ranking of VLMs on the target task). Therefore, our experiments mainly focus on zero-shot image classification tasks.

  2. Secondly, our SWAB is not specifically designed for image classification tasks. For the Modality Gap, our method aims to reduce the distribution differences between different modal data. For the Capacity Gap, our goal is to use the model's ranking in related historical tasks to predict its ranking in new target tasks. These designs are not tied to image classification and can be extended to other zero-shot tasks. For example, in zero-shot image-text retrieval tasks, we can use our SWAB method to better utilize text samples as substitutes for image samples for model evaluation.

  3. Thirdly, zero-shot image classification tasks are highly representative applications for VLMs and are among the most important and valuable tasks for VLMs. We believe that the excellent model selection results of our method in this task already demonstrate the significance and effectiveness of our approach.

[1] Zohar et al. Lovm: Language-only vision model selection. NeurIPS 2023.

[2] Radford et al. Learning transferable visual models from natural language supervision. ICML 2021

[3] Li et al. Align before fuse: Vision and language representation learning with momentum distillation. NeurIPS 2021.

[4] Li et al. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. ICML 2022.

[5] Zhang et al. Diagnosing and Rectifying Vision Models using Language. ICLR 2023.

Dear Reviewer omtt,

We hope this message finds you well. I am reaching out to kindly request your prompt response to confirm whether our responses adequately address your queries. We sincerely thank you for your time and effort during this discussion period. Your timely feedback is greatly appreciated.

Dear Reviewer omtt,

We have responded to your comments in detail. As the discussion period will end in less than one day, we would like to ask whether there are any additional concerns or questions we could address.

Thanks very much for your effort!

Best regards,

Authors

Dear reviewer omtt,

We thank you sincerely for your time and effort in reviewing our manuscript and providing valuable suggestions. As the author's response period is nearing its conclusion, we kindly remind you to review our response.

We have provided detailed responses to your questions and hope that they adequately address your concerns. If you need further clarification or have any other questions, please feel free to discuss them with us! We are more than willing to continue our communication with you. We would greatly appreciate it if you would update the rating by taking our responses into account.

Dear Reviewer omtt,

We have responded to your comments in detail. As the discussion period will end in less than 10 hours, we would like to ask whether there are any additional concerns or questions we could address.

Thanks very much for your effort!

Best regards,

Authors

We appreciate the reviewers' comments. Below are our responses:

• Q1: How many image samples were used in the experiment.

• A1: We followed LOVM's approach [1] to use 23 datasets. Each of the 23 datasets was used as a test dataset in turn, with the remaining 22 datasets serving as open-source datasets. Therefore, the size of the open-source datasets varied for different tasks. The largest open-source dataset contained 497,123 images, the smallest contained 389,080 images, and the average number of images was 476,189.

  It is important to emphasize that the previous SoTA method, ModelGPT [1], also used the same open-source datasets as we did.

• Q2: Provide the accuracy vs number of open-source image samples.

• A2: We conducted additional experiments by randomly sampling images from the original open-source datasets at different ratios to create new open-source datasets and applied our method based on them. We use the same sampling ratio for each category. We repeated the experiment 5 times and show the averaged the results. We found that the variation among different experimental results was small. The results are as follows:

  Data Ratio	40%	60%	80%	100%
  $R_5$	0.508	0.519	0.525	0.534
  $\tau$	0.247	0.256	0.253	0.260
  $R_5$ + $\tau$	0.755	0.775	0.778	0.794

  The result shows that our method is robust to the number of data in the open-source datasets used. This is because the VLM's average gap vector of each category and the VLM's ranking of each category are robust to data volume. As shown in Table 5 in our paper, gap vectors for different samples within the same category are consistent. Additionally, reducing the dataset size has minimal impact on model rankings.

• Q3: Add the ablation study for no learning based SWAB.

• A3: Firstly, we emphasize that our method focuses on bridging the modality gap and capacity gap in VLM selection. In achieving these goals, we do not introduce any learning process. In SWAB, only the Rank Model in the third step depicted in Figure 3 of our paper requires training. And this Rank Model is introduced by the ModelGPT we follow. We use our SWAB to optimize the ModelGPT's capabilities by bridging the modality gap.

  Regarding the reviewers' concerns about the improvements of our method over non-learning based methods, the corresponding results are shown in Table 3 of our paper's main text and Figure 5 in the appendix. The results shown in both table 3 and figure 5 do not use the rank model, and therefore, no training process is introduced:

  • In Table 3 of the main text, We demonstrated the performance improvement in selecting the VLM by using the weighted average of VLM's ranking on categories of open-source dataset with SWAB to bridge the capacity gap, compared to using the average VLM's ranking of various categories of open-source data directly.
  • In Figure 5 of the appendix, we separately show the top-1 classification accuracy calculated on text samples before and after using SWAB to bridge the modality gap between text and image data. This demonstrates a better rank consistency between the text top-1 classification accuracy of VLMs and actual classification accuracy of VLMs after using our SWAB.

• Q4: Why adding Gaussian noise to the target generated embedding?

• A4: We added Gaussian noise to simulate the variance in image samples because different images can correspond to the same caption. For example, in the text "A dog running on the grass," the dog in the images can vary in breed. We add Gaussian noise to the features of text samples to simulate the diversity of images. We found that adding Gaussian noise improves the performance of our method. Some existing works [1, 2] have also adopted similar approaches.

• Q5: How to select weighted parameter in Eq.9.

• A5: $\alpha$ is a hyperparameter. We used the same alpha value of 0.45 for all experiments.

• Q6: Test the open-source LLM like Llama2 to generate the Auxiliary texts.

• A6: We used Llama 2 to generate the corresponding text and found that our method performs excellently with texts generated by different LLMs. We attribute this to the fact that generating category-specific diverse texts is a relatively simple task for LLMs, and common LLMs possess this capability. | Metrics | $R_5$ | $\tau$ | $R_5 + \tau$ | |---------|------|------|----------| | Llama2 | 0.527| 0.254| 0.781 |

[1] Zohar, Orr, et al. "Lovm: Language-only vision model selection." NeurIPS 2023.

[2] Zhang, Yuhui, et al. "Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data." ICLR 2024.

Dear Reviewer NUg2,

We hope this message finds you well. I am reaching out to kindly request your prompt response to confirm whether our responses adequately address your queries. We sincerely thank you for your time and effort during this discussion period. Your timely feedback is greatly appreciated.

Thank you for your detailed response and for addressing my concerns. However, I have a few more questions. Is the parameter $\alpha$ sensitive to different datasets, and what is the best strategy for selecting it? Additionally, could you provide some analysis on this? Another question for clarification: Regarding the experiment setting in Q2, is the performance measured on just one specific test dataset, or is it an average across all 23 test datasets?

Dear Reviewer NUg2,

We have responded to your comments in detail. As the discussion period will end in less than one day, we would like to ask whether there are any additional concerns or questions we could address.

Thanks very much for your effort!

Best regards,

Authors

We greatly appreciate the reviewers' recognition of our work. Below are our responses to the relevant questions:

• Q1: The performance of ModelGPT is slightly inconsistent with the results in its paper.

• A1: This is because we conducted multiple repeated experiments using different random seeds. ModelGPT added Gaussian noise to perturb the features of the text samples. To enhance the reliability of the results, we used 10 different random seeds for repeated experiments and reported the mean and standard deviation of the results. Since the original ModelGPT paper did not perform repeated experiments, our results show slight differences compared to their paper.

• Q2: The method of setting the $\alpha$ value in formula 9.

• A2: $\alpha$ is a hyperparameter. We used the same alpha value of 0.45 for all experiments.

Dear Reviewer qHTS,

We hope this message finds you well. I am reaching out to kindly request your prompt response to confirm whether our responses adequately address your queries. We sincerely thank you for your time and effort during this discussion period. Your timely feedback is greatly appreciated.