Vision-Language Model Selection and Reuse for Downstream Adaptation

The process of constructing the reference dataset, profiling the target dataset, and conducting multi-model inference, replying on a combination of language generation models, embedding models, and similarity computation, is complex and time-consuming. As can be seen in Tab.2, a suboptimal choice of k can lead to a decline in results. Plus the details of this experiment are missing, it is important to question how robust the selection of k=3 is across 17 different downstream tasks. If the value of k is uncertain unless given a validation split, it becomes challenging to demonstrate how effective the procedures could be in practical applications.

The proposed method is straightforward and, for the most part, clearly explained. Two minor suggestions:

1. The logic from eq6 to eq7 is hard to follow.
2. \mathcal{T}_m(D) should be \mathcal{T}_m(d) in eq2.

The submission has not been well validated, and analysis is weak and not insightful enough. Besides above mentioned weakness,

1. important baselines are missing. For example, since ensemble typically enhances performance and random sampling can ensure diversity in recognizing different classes, it is crucial to conduct random ensemble with majority voting / prob. distribution average / etc as ablation study, other than the parameter analysis presented in Section 5.3.
2. how do the randomly selection reference images affect the robustness of the proposed method?
3. how do specific downstream tasks correlate with different model candidates? For example, why does the method show significant improvement on datasets DMLab, RESISC45 and FER2013, while performing at a comparable level on other datasets?

The proposed method, while simple and straightforward, lacks convincing validation and practical applicability. The method's effectiveness is limited even after model ensemble. It also raises important questions about the robustness and practicality of its approach, especially considering the complexity and uncertainty involved in key processes like target benchmark profiling, model selection and dataset correspondence. These factors suggest that while the paper addresses a relevant and meaningful task, its contributions may not significantly advance the field without further validation and refinement.

Given that the authors present the proposed paradigm as one of their main contribution, it is essential to thoroughly evaluate its effectiveness compared to other training-free methods for VLMs, such as but not limited to test-time adaptation.

The proposed method seems to be work. To evaluate the performance of the method, the authors introduce a semantic graph with labels and corresponding images to pre-test the performance of each model. Subsequently, when reusing each model, the authors propose a model identification strategy to locate each model in an embedding space and ensemble models for subsequent tasks. The overall framework has potential to demonstrate the effectiveness of a collection of models. The benchmark and evaluation in this paper are valid and can reflect the performance of each method.

N/A

Both the experimental design and analyses are comprehensive. The experiments are conducted on multiple datasets with up-to-date comparison methods. Moreover, the authors give scalability and ablation studies to further demonstrate the effectiveness of proposed methods.

The method presented in this paper is significant, as it can be utilized to manage existing Vision-Language Models (VLMs) to address new downstream tasks with improved performance, rather than constructing new VLMs, which is costly. These techniques can assist platforms like HuggingFace in better organizing models.

No necessary reference is omitted.

To achieve effective reuse of VLMs, the paper proposes Model Label Learning (MLL), an efficient approach for selecting and reusing pre-trained Vision-Language Models (VLMs) for downstream tasks. Specifically, the framework consists of three modules: (1) Model Labeling, which assigns labels to VLMs based on their capabilities; (2) Model Selection, which matches these labels to task requirements; and (3) Model Reuse, which employs an ensemble of selected models. The proposed framework is reasonable for addressing this problem.

Additionally, a large-scale benchmark, including 49 VLMs and 17 datasets, is introduced to evaluate the effectiveness of MLL.

NA

The paper presents comprehensive experiments involving 49 VLMs and 17 target task datasets, along with sufficient analysis. The experimental results demonstrate the effectiveness of the proposed method in selecting and reusing VLMs.

In addition to the experiments mentioned above, the authors should provide more experimental results to further validate the proposed method. For example, for each target dataset, the highest performance achieved by any model in the model hub should be included as a reference. This would help assess the effectiveness of the proposed method in selecting models.

Unlike other VLM studies, this paper explores a practical VLM reuse problem, aiming to effectively reuse existing models to improve performance on target datasets.

NA

The proposed methods and evaluation criteria are generally sensible and appropriate for the problem of VLM selection and reuse.

The MLL method with its three modules is a logical approach to address the VLM selection problem. Semantic labeling provides a structuredway to understand VLM capabilities without task-specific evaluation. Semantic matching for selection is an efficient alternative to exhaustive search. Ensemble reuse leverages the strengths of multiple selected models.

The introduced benchmark of 49 VLMs and 17 downstream datasets is a strong evaluation criterion, covering a range of domains and tasks relevant to VLMs. The authors propose to use accuracy and F1-score, which are standard metrics for the evaluated tasks. Comparison against INB and ModelGPT provides relevant baselines.

There are no explicit theoretical claims made in the submission excerpt that require proof checking. The paper is primarily methodologically and empirically driven. Claims of efficiency and scalability are based on the design of the method and empirical observations, not formal theoretical derivations.

The experimental designs and analyses are sound in principle but require more detailed reporting and validation.

Soundness

Using INB and ModelGPT as baselines is appropriate and allows for comparison to a simple baseline and a relevant existing VLM selection method. The use of a diverse benchmark is a strength. The ablation studies on $\alpha$ (table 3) and model count $k$ (table 4) demonstrate an effort to understand the impact of key hyperparameters and design choices.

Issues and areas for improvements:

1. statistical significance: as aforementioned, this submission lacks mention of significance test to validate the observed performance differences.
2. more detailed result: while average performance is reported, the readers might also wonder: are the improvements consistent across all datasets? Also, are there specific datasets or task types where MLL excels or underperforms?
3. computational cost: the analysis of the cost of model labeling should also be reported.

This paper appears to be relating to three research areas:

VLM: The paper directly relates to a challenge in VLMs - how to efficiently and effectively utilize the increasing number of pre-trained models.

Model Selection: The paper also relates to literature on model selection, particularly for pre-trained models. It acknowledges and contrasts with existing methods like NCE, LEEP, LogME, and ModelGPT.

Learnware/Model Hubs: The paper explicitly connects to the "learnware" paradigm and the concept of model hubs. It builds upon the idea of model specification and efficient model selection in heterogeneous model repositories. MLL can be viewed as a specific instantiation of learnware principles tailored for VLMs.

Primarily I don't think there are significant missing of essential literature; however, to strengthen the literature context and ensure comprehensiveness, the authors might consider discussing and citing works in the following more specific areas:

• semantic similarity measures: as MLL relies on semantic similarity measure for comparing captions, discussing relevant papers on semantic textual similarity and different metrics would be beneficial.
• ensemble methods for VLMs: If the ensemble reuse module could be extended to more ensembling techniques, making a discussion on relevant works on ensemble methods, would also be important.