Can We Predict Performance of Large Models across Vision-Language Tasks?

We greatly appreciate your time and effort in reviewing our paper. We are pleased that you acknowledged our extensive evaluation and our enhancements to address the sparse-data issue.

We still need to directly evaluate.

We agree with you that direct evaluation is very helpful and important. However, it is expensive to comprehensively evaluate LVLMs. Zhang et al. [3] reports that it takes hundreds of hours to evaluate one model on around 50 tasks in LMMs-Eval, and evaluation even exceeds 1,400 hours on models of 100B parameters or more. With the growth of LVLM benchmarks and models, the evaluation will be more costly.

Our framework can benefit LVLM evaluation in two ways.

• First, it can reduce unnecessary evaluation, given a limited computational budget in practice.
• Second, it can prioritize the direct evaluation experiments by using our uncertainty-based active evaluation method.

Thus, we respectfully argue that our framework is useful and valuable in practice, which is acknowledged by Reviewers uZXr, VvNb, and Q3Aj.

Reliability of reducing test sets and how to trust our framework.

We first reference related works to support our paper and then highlight the role of uncertainty estimation in our framework.

Recent works select a coreset of samples from a large benchmark for evaluating LLMs [1, 2] or LVLMs [3, 4]. The performance of a specific model on the coreset is used to estimate its performance on the full benchmark. Thus, it is possible to predict model performance while maintaining reliability.

Moreover, our framework provides uncertainty in performance prediction, which is correlated with the actual absolute errors in Figure 4. The estimated uncertainties can help identify wrong predictions.

We hope that our response can addresses your concern.

Our references:

Efficient LLM evaluation:

[1] tinyBenchmarks: evaluating LLMs with fewer examples. ICML 2024.

[2] Efficient benchmarking (of language models). NAACL 2024.

Efficient LVLM evaluation:

[3] LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models. https://arxiv.org/abs/2407.12772.

[4] LIME: Less Is More for MLLM Evaluation. https://arxiv.org/abs/2409.06851.

We sincerely appreciate your time and effort in reviewing our paper. We summarize your main points and try to address them one by one.

Contributions of our paper, compared to tinyBenchmarks and Efficient Benchmarking.

We would like to highlight that our framework is different from previous works [1-4].

Different problem settings. We predict unknown model performance scores based on known ones across benchmarks and models, while tinyBenchmarks and Efficient benchmarking focus on reducing the size of a benchmark.

For example, let's evaluate models A and B on the SEED and MMMU benchmarks. We are thinking about whether we could use the performance of A on MMMU to predict that of A on SEED, or use the performance of B on MMMU to predict that of A on MMMU. Our goal is to reduce the total number of evaluations. While related works [1-4] aim to reduce the size of SEED and MMMU benchmarks, making each-time evaluation more efficient.

Different methods. We formulate our problem as a matrix completion task and use PMF with MCMC to solve it, while previous works usually rely on coreset-selection methods.

Compared to LLMs, LVLMs have less standardized evaluation pipelines and inference methods.

We are sorry that we are not sure what you mean by "LVLMs have less standardized evaluation pipelines". There are great benchmarks such as SEED, MMBench, MMMU, and POPE. We have included these benchmarks in our experiments. There are also standardized and generalized evaluation pipelines like LMMs-eval and VLMEvalKit. We build our study on these pipelines. We could provide a more specific answer if you would like to provide more explanation.

Narrow contribution in the LVLM field

We respectfully highlight that we conduct a comprehensive evaluation of 108 models on 176 datasets, covering a wide range of tasks and benchmarks. This systematic evaluation can provide a foundation for future research.

Moreover, in the Dicussion section of our paper, we conduct further analysis on the correlation of model performances, what are the effects of vision encoders in LVLM on benchmarking, and which LVLMs or benchmark results are more informative to performance prediction.

If you have any more questions or suggestions, we are happy to discuss them with you.

Thank you for your feedback! Below, we address your concern in detail.

Combine few-shot evaluation with our method.

Previous studies ("few-shot evaluation" in Reviewer dktp's comment) select a small set of representative samples (coreset) in a benchmark and evaluate models on the coreset. Our work uses known model performance from different benchmarks or models for performance prediction. Our method is complementary to these existing approaches and can be combined with their few-shot evaluation.

In experiments, we explore two combination methods:

(1) Avg. Simply get the average prediction of few-shot evaluation and PMF;

(2) Unc. Use uncertainties from MCMC to combine few-shot evaluation and PMF predictions. In short, when PMF is confident, we mainly rely on using known performance for prediction. Otherwise, the prediction is more dependent on few-shot evaluations

Following LMMs-Eval, we use CLIP to generate embeddings for images and BGE-M3 for text, and concatenates them to create the final embeddings. Based on sample embeddings, we use random, Herding, and K-Center Greedy [5] to select core samples.

The following table presents the average RMSE values of 3 experiments, demonstrating the effectiveness of our approach.

[5] DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning. https://arxiv.org/abs/2204.08499.

Incorporate evaluation cost into the framework.

Thank you for the suggestion! It is very interesting to incorporate evaluation cost into our framework.

A straightforward way is to implement a cost-aware heuristic function in active evaluation. Instead of using only uncertainty, we assign a score to each model-dataset pair, $score = f(uncertainty, cost)$. The model-dataset pairs with higher scores will be prioritized for evaluation. Some possible functions are $f(a, b)=a^\gamma b^{1-\gamma}, f(a, b)=a + \gamma b$.

It is not easy to measure evaluation cost. Different models may have different acceleration techniques, some models are API-only, and evaluation samples may have various context lengths. Here, for each dataset, we simply use the number of samples / the total number of samples of all datasets to approximate the evaluation cost, and conduct a preliminary experiment.

We follow the setting recommended by Reviewer Q3Aj. In short, we use 20% performance scores for initial training, 60% as the pool set, and 20% for testing. PMF is trained on the initial 20% data. In each iteration, we use a method to order the pool set and select the top model-dataset pairs. We retrain PMF in each iteration with extra data from pool set, and evaluate the model on the test set.

The following table shows the RMSE Improvement (%) on the test set, where each column is the number of extra evaluated samples. If a method chooses more samples in each iteration, we fit the result into the closest column. As seen, our methods show significant improvement over Random.

Interestingly, we find that, the error of performance prediction is mainly caused by small datasets, which may have large variance in performance due to limited sample size. Thus, evaluate models on these highly-uncertain but low-cost datasets may be the best for performance prediction. It is worth exploring more practical design and we note that our framework is extensible for that.

Not a novel contribution

Thank you for acknowledging we are addressing an important problem. We respectfully highlight that the main contributions of our paper are: (1) formulating the problem of LVLM performance prediction based on known performance scores; and (2) connecting well-established algorithms to the novel application and demonstrating their effectiveness. Our main focus is the new application, instead of introducing technical contributions on a previous problem.

Incorporated VLM specific ideas or benchmark specific stuff.

Thank you! We respectfully highlight that we incorporate model and dataset profiles into PMF (Section 3.5). For models, we include features such as the number of parameters in the LLM backbone, vision encoder type, and the LVLM family. Additionally, we explore three different approaches to generate these latent representations and cluster the LVLM benchmarks. These profiles may consider VLM- or benchmark-specific stuff as you expected, are extensible for extra model or dataset information, and improve the accuracy in performance prediction.

We sincerely appreciate your time and effort in reviewing our paper. We are pleased that you acknowledged the solid foundation, effectiveness, practicality, and scalability of our paper.

The paper tackles an important problem but the novelty is somewhat limited.

Thank you for acknowledging we are addressing an important problem. We respectfully highlight the main contributions of our paper are (1) we formulate the problem of LVLM performance prediction based on known performance scores; and (2) we connect well-established algorithms to the novel application and show their effectiveness. Our main focus is the new application, instead of introducing technical contributions on a previous problem.

How Bayesian PMF differs from standard PMF in practical terms?

In real-world situations, we may only have limited performance data about an LVLM or benchmark for training PMF. For example, if OpenAI released GPT-5 yesterday, we might know its performance on only 5 benchmarks. In cases like this, Bayesian PMF can predict performance more accurately than the standard PMF model, which is shown in Figure 5(A).

The reason is that Bayesian PMF defines distributions over the parameters of prior distributions, known as hyperpriors. The hyperpriors work similarly to regularization terms in a loss function and improve model performance when there is limited data available. As we gather more data, the advantage of using hyperpriors over a standard model becomes less noticeable.

How does incorporating an LKJ prior impact the predictions in practice?

Using an LKJ prior is primarily for computational reasons, rather than improving predictions. We will clarify this in the revised paper.

In short, Wishart distribution models the distribution of covariance matrices. It has two main issues during the sampling process.

• First, it requires the sampled matrices to be both positive-definite and symmetric. The probability of generating a valid sample by randomly changing elements is close to zero.
• Second, the distribution has a very heavy tail, which poses many challenges for simple sampling methods.

Instead, we use the LKJ correlation prior and an Exponential prior, which are computationally advantageous.

Our references:

This is suggested by PyMC Official Documentation.

LKJ Cholesky Covariance Priors for Multivariate Normal Models. https://www.pymc.io/projects/examples/en/latest/howto/LKJ.html

There are also discussions on practical issues about the Wishart distribution vs LKJ on coding forums like GitHub and StackExchange.

https://github.com/pymc-devs/pymc3/issues/538#issuecomment-94153586

Ablation study to better quantify the contribution of each component.

We conduct a detailed ablation study as suggested. The table below shows each component’s contribution when the training performance scores are sparse. Here, PMF methods model each metric separately. All results are the average RMSE over 10 experiments, with lower values indicating better performance.

Some results are already presented in Table 1 and Figure 5 in the paper. We will include the detailed results in the supplementary materials.

Gap between the uncertainty-based active evaluation and the oracle method.

We implement a canonical active learning evaluation setup, as suggested by Reviewer Q3Aj. In short, we use 20% performance scores for initial training, 60% as the pool set, and 20% for testing. PMF is trained on the initial 20% data. In each iteration, we use Random / Active (Ours) / Oracle methods to order the pool set and select the top model-dataset pairs. We retrain PMF in each iteration with extra data from pool set, and evaluate the model on the test set.

The following table reports the improvement on the test set as we acquire more evaluations in the pool set.

As shown in the table, our method demonstrates significant improvement over Random and approaches or even exceeds the performance of Oracle.

Additionally, as suggested by Reviewer uZXr, we also explore cost-aware active evaluation and demonstrate the advantages of our methods. We kindly refer you to our reply to Reviewer uZXr.

We greatly appreciate your time and effort in reviewing our paper. We are pleased that you acknowledged the strong motivation, elegance, effectiveness, clear writing, and comprehensive evaluation of our work.

The limited set of baseline matrix completion methods.

As you suggested, we evaluate Deep Matrix Factorization (DMF) [1] and Graph Convolutional Matrix Completion (GCMC) [2] for a more comprehensive comparison.

For DMF, we use MSE loss and the Adam optimizer. The learning rate is 1e-3 and the batch size is 256. The embedding dimension of each user or item is 10, which is the same for PMF. We train DMF for 200 epochs and the result of the best epoch is reported.

For GCMC, we refer to the GitHub implementation (https://github.com/riannevdberg/gc-mc). Dropout ratio is 0.7, learning rate is 0.01, hidden units are [500, 75] in 1st and 2nd layers, accumulation method is "stack", the number of basis functions is 2, and the model is trained for 200 epochs. We note that there are two main issues when using GCMC:

• GCMC handles discrete rating levels and treats each rating level as a separate class (see Section 2.3 [2]), which is not suitable for our setting, because we use continuous ratings like the BART scores. To address this, we only use LVLM benchmarks with accuracy as the main metric, and discretize accuracy into 101 classes, i.e., {0, ..., 100}.
• When training data is sparse, some classes (for example, accuracy = 67) do not occur in training set, leading to running errors in the code.

The table below summarizes the average results from 10 experiments. As shown, PMF demonstrates superior performance on our dataset compared to DMF and GCMC.

We notice that matrix completion methods are commonly applied in recommender systems, where there are usually thousands of users and items, with millions of samples, such as the Movielens dataset. But in our setting, we have 108 models, 176 datasets, and 19K samples. Thus, we build our method on a simple but strong baseline, PMF, rather than neural networks, which are possibly more data-hungry.

A more canonical active learning evaluation setup.

Thank you for the suggestion! We use 20% data for initial training, 60% as the pool set, and 20% for testing. The following table reports the improvement on the test set as we acquire more evaluations in the pool set.

As shown in the table, our method shows significant improvement over Random, and is close to Oracle. Interestingly, when we acquire around half of the pool set, the model shows better performance than using the entire pool set. We will use this setup and update related results in our paper.

Extension to transductive active learning.

Thank you! It is very interesting to explore transductive active learning within our framework. In practice, we might ask questions like, "What evaluation experiments can best inform the performance of models that use CLIP as vision encoders?" or "Which experiments provide the most useful information for improving our own models?" In such cases, instead of looking at uncertainties across all predictions, it could be more helpful to measure the information gain to a specific model or dataset.

An intuitive approach would be to integrate the expected predictive information gain (EPIG) method proposed by Bickford-Smith et al. [4] into our framework. This idea diverges from our current focus and would be better suited for future work.

Thanks for your thorough response addressing my concerns on lack of deep baselines, and adding the active learning experiment in the more canonical setting. I have raised my score from 6 to 8.