Can We Predict Performance of Large Models across Vision-Language Tasks?

Novelty of the paper

Thank you for your question! We would like to highlight three main contributions that reflect the novelty of our work.

First, we propose and formulate the problem of LVLM performance prediction based on known performance scores. Previous works on efficient evaluation need to select a coreset within each dataset or use an unlabeled test set to estimate model performance. Instead, we explore a new direction and we show that our approach is complementary to coreset-based methods in Section 5.1.

Second, we introduce model profiles and dataset profiles in Section 3.5 to enhance the prediction accuracy. For model profiles, we include features such as the number of parameters in the LLM backbone, the vision encoder type, and the LVLM family. For dataset profiles, we cluster datasets based on latent representations obtained from various models and get one-hot encoded dataset profiles. We validate three different approaches to generate these latent representations:

Third, we propose an uncertainty-aware active evaluation pipeline for efficient model evaluation. As shown in Fig. 4, our uncertainty-aware evaluation consistently outperforms the random baseline under a fixed evaluation budget, especially when the budget is limited. Besides, the estimated uncertainties are correlated with the actual absolute errors, indicating the reliability of our confidence estimates.

We hope that our paper can inspire future works, reduce redundancy in evaluation of large models, and contribute to more efficient development of LLMs and LVLMs.

Relationship between the proposed method with the evaluation of LLMs. Why do you focus on the evaluation of LLMs?

Thank you for your question! We focus on LLMs and LVLMs for two main reasons. First, the evaluation of these large models is very expensive, and the number of models and datasets are growing rapidly. Second, LLMs can handle a wide variety of tasks within a single model, making it very interesting to explore the correlations of their ablities across different tasks.

To reduce the evaluation costs, our paper explores a question: Can we predict unknown model performance scores based on known ones?

To answer this, we draw insipration from recommender systems and adpot PMF to our question. PMF is a well-established algorithm for matrix completion. In our case, we construct a sparse performance matrix R, where each entry represents the performance score of a model on a dataset. By applying PMF, we aim to complete the performance matrix, that is, predict unknown scores. This makes PMF a natural choice for our problem.

While our paper focuses on LVLMs, the proposed idea and methods are general and can be applied to other types of models and tasks as well.

As an extension, we reference a related work [1], which ranks samples for efficient evaluation in image classification models. They introduce the Lifelong-ImageNet benchmark, which contains 1.98M test samples and results from 167 models.

To extend our method, we validate PMF with a Sigmoid output layer to predict sample-level model accuracy (0 or 1) on Lifelong-ImageNet. Specifically, we use 10% of the performance data for training and 90% for testing. Due to the large scale of this dataset, MCMC becomes pretty slow. Thus, we use the L-BFGS-B optimization algorithm to get a maximum a posteriori estimate. The results are shown as follows.

*[1] explores a different setting to ours, so the numbers here are only for reference.

More meticulous design may futher improve the performance of PMF. However, such improvements are beyond the current scope of our paper and are left for future work.

Typos and mistakes

Thank you! We will carefully refine our paper and correct the typos.

[1] Efficient Lifelong Model Evaluation in an Era of Rapid Progress. NeurIPS 2024.

The computational cost

Q: Evaluation cost with the newest LMMs-Eval implementation.

Thank you for your suggestion! We would like to clarify vllm was integrated into LMMs-Eval after our submission. We did not intend to exaggerate anything.

In the rebuttal, we do not have enough time to rerun all our evaluations. Instead, we evaluate a representative model, Qwen2.5-VL-Instruct based on LMMs-Eval with 1 A100 GPU.

*Video loading and preprocessing are the botteneck.

As shown in the first columns, although vllm significantly accelerate the evaluation, the computational cost is still high. Assuming that we are comparing 10 7B models on 10 similar-scale benchmarks, the evaluation will take around 21.7 hours. Besides, in our experiments, vllm usually leads to slight performance decrease.

LMMs-Lite (a coreset method) significantly reduces the evaluation cost, indicating reducing evaluation cost is still valuable in practice. As shown in Section 5.1, coreset methods may get inaccurate results, while our method improves them.

Q. The cost will be less significant in the future

We agree that, with the development of software and hardware, evaluating old models on the old benchmarks is becoming cheaper. However, evaluation remains costly for the two main reasons.

First, the growing number of models and benchmarks significantly increases the overall evaluation burden. After our submission, just from January to March, there are already several new models (e.g., the DeepSeek series, Qwen2.5 variants, MM-Eureka) and benchmarks (e.g., MathGlance, Mobile-MMLU, MathFlow, MMDT, MapBench).

Second, the trend toward larger models, video LLMs, and test-time scaling techniques further raise evaluation costs, as illustrated in our table.

We need real performance numbers to write a paper/report instead of an estimated number.

Thank you! We totally agree. Not only our paper, previous studies on predicting model performances, e.g., TinyBenchmarks and model auto-evaluation, also suffer from this problem.

However, our method is still valuable during model development. For instance, when exploring the optimal video LLM design [1], the researchers evaluate their models on a reduced set of benchmarks to guide design decisions. Only at the final stage, they evaluate the model on the full benchmarks to report real performance numbers. In this case, the reseachers could use our method to reduce the evaluation cost in model development.

OOD generalization of the method

Thank you! The question is also raised by Reviewers Yhor and dJrE. We add new models and new datasets into our pool.

New models. Qwen2-VL-Instruct (2B, 7B), Qwen2.5-VL-Instruct (3B, 7B, 32B), DeepSeek-VL (tiny, small)

New datasets. MathVision, EMMA, Video-MME, LongVideoBench

The averaged RMSEs are shown as follows.

If we only know 20% performance of new models and new datasets

If we know 50% performance of new models and new datasets

Our method shows better generalization compared to the baselines.

Different models may require different optimal prompts

In the supplementary material (Section B.5.), we apply different evaluation settings to LLaVA and explore two ways to extend our methods. The results show that our framework can predict model performance under different evaluation settings.

The clear gain of the GPT-4 series

Thank you! We separate models to two groups: (1) Train w/ GPT, such as LLaVA and Cambrain. (2) Train w/o GPT, such as BLIP-2 and InstructBLIP. We run 10 experiments to get the averaged results. For "Train w GPT", adding evaluation results of GPT-4o decreases 0.029 (2.48%) RMSE of PMF, while for "Train w/o GPT", it decreases 0.022 (2.04%) RMSE. Thus, distilling knowledge from GPT may lead to similar strengths and weaknesses, which needs further exploration.

Typos

Thank you!

Explanation to Figure (6)B

In Fig. 6(B), the tasks in tail do not have the same RMSE improvements, but their values are relatively close. We observe that, these tasks show smaller performance gaps across models, so adding them brings limited RMSE improvement.

[1] Apollo: An Exploration of Video Understanding in Large Multimodal Models. ArXiv.

Metric Validity

Thank you for your suggestion! We include the following ranking-based metrics.

Spearman’s rank correlation.

Kendall rank correlation.

Precision@K. The proportion of the predicted top 1 model that fall within the top K positions of the ground-truth ranking. For example, our method predicts LLaVA-1.5 is the best on the task and it will be correct if LLaVA-1.5 is within top 5 of the ground-truth ranking for Precision@5.

Recall@K. The proportion of the ground-truth best ones that is correctly retrieved within our top K predicted results. For instance, if LLaVA-1.5 is the best model, it must be within the top 3 predictions of our method for Recall@3.

If we know 80% performance data for training

Baseline methods typically achieve very high precision but low recall, while our methods provide more balanced precision and recall, as well as improved rank correlation.

"Unc 50%" and "Unc 30%" mean keeping the 50% or 30% most confident predictions based on our estimated uncertainty, which further improve the estimation accuracy. But note that this may lead to fewer predicted results.

If we know 20% performance data for training

Our method still outperforms baselines.

Due to the space constraints, we omit additional results. We will update our paper accordingly.

Generalizability to New Benchmarks

Thank you! The question is also raised by Reviewers Yhor and mCcw.

We add new models and new datasets.

New models. Qwen2-VL-Instruct (2B, 7B), Qwen2.5-VL-Instruct (3B, 7B, 32B), DeepSeek-VL (tiny, small)

New datasets. MathVision, EMMA, Video-MME, LongVideoBench

The averaged RMSEs are shown as follows.

If we only know 20% performance of new models and new datasets

If we know 50% performance of new models and new datasets

Our method shows better generalization compared to the baselines. We find that generalizing to new datasets is easier than to new models. This is probably because new datasets are often very challenging, leading to generally low model performance and small RMSEs. In contrast, new models often have novel designs and remarkable improvements. This makes generalization to unseen models more difficult and needs further exploration.

How to interpret our results?

There are two possible perspectives.

Benchmarks. There is redundency in existing benchmarks, as also reported by [1]. Multiple benchmarks may test similar skills like math. Besides, there might be some correlation across different tasks. For instance, [2] reports that an LVLM with a stronger LLM can achieve consistently better performance on different benchmarks.

Models. The similarity in model architectures and training may also contribute to the correlation between performance. For example, if two models use the same vision encoder, e.g., CLIP, they may have similar failure cases [3]. Besides, as commented by Reviewer mCcw, many models use training data generated from GPT-4V, possibly resulting in similar strengths and weaknesses.

We will update our paper to include the discussion.

How do users rely on our predictions?

Thank you! As shown above, the model rankings from our methods have high correlation to the ground truth. Even with very sparse training data, our methods remain more reliable than guessing based on average scores, a strategy that mimicking what a human might do. Moreover, our estimated uncertainty can further support users to make decisions. Users can focus on predictions with higher uncertainty, conduct additional evaluations, and make better model selections.

[1] Redundancy Principles for MLLMs Benchmarks. ArXiv.

[2] LLaVA-NeXT: What Else Influences Visual Instruction Tuning Beyond Data?. Blog.

[3] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. CVPR2024.

Explanation to our method

Q: Why to formulate it as a matrix prediction problem? What is the underlying principle or intuition?

We are inspired by recommender systems. Imagine we are recommending movies to users: there are many users and many movies, but each user only rates a few movies. We can build a matrix where each row represents a user, each column is a movie, and each element is a rating from a user to a movie. We observe some elements in the matrix and want to estimate the unknown ones for making recommendation. This suits a matrix completion task and PMF is an effective way to solve that. In our case, we can see LVLMs as “users” while benchmarks as “movies”. We observe some performance scores of LVLMs on some benchmarks, and want to predict the unobserved ones. Thus, we formulate our problem similarly and solve it with PMF.

Q: Is it specifically applicable to LVLM?

While our paper focuses on LVLM evaluation, the methods are general and can be applied to other models and tasks as well.

Q: Do LVLM models and benchmarks satisfy the Gaussian distribution condition?

The Gaussian distribution assumption primarily serves as quadratic regularization terms in the objective function [1], which can alleviate the overfitting problem. It does not require that the latent distributions must be Gaussian. To verify this, we apply Kolmogorov-Smirnov test to check if the learned latents follow a Gaussian distribution. The p-values are all below 0.05, indicating they are not actually Gaussian.

How can the framework be extended to handle different evaluation settings

Thank you for your question! We show some preliminary experiments in the supplementary material (Section B.5.). Specifically, we explore two ways to extend our methods.

Additional profile. The other method is to encode evaluation settings as extra profile into PMF.

Additional models. A straightforward way is to treat a model under different evaluation settings as different models, such as "LLaVA (Chain-of-Thought)" and "LLaVA (Beam Search)".

We evaluate LLaVA-v1.5-7B on the 27 tasks in SEED-2, with the following various evaluation settings.

Image input. (1) Default: use clean images, or (2) add Gaussian noise into the images.

Prompt. (1) Default: prompt the model to choose option directly, (2) provide no hint, or (3) use the Chain-of-Thought (CoT) prompt.

Model decoding. (1) Default: greedy decoding, (2) sampling with temperature = 0.2, (3) sampling with temperature = 0.5, or (4) beam search with temperature = 0.2 and the number of beams = 10.

We add the results under different evaluation settings into our framework and apply PMF. The estimation accuracy is shown as follows, indicating that our framework can predict model performance under different evaluation settings.

Test Ratio: 20% (More results are in Section B.5.)

The generalization capabilities of the framework

Thank you! The question is also raised by Reviewers dJrE and mCcw. To validate the generalization ability of the framework, we add new models and new datasets into our pool.

New models. Qwen2-VL-Instruct (2B, 7B), Qwen2.5-VL-Instruct (3B, 7B, 32B), DeepSeek-VL (tiny, small)

New datasets. MathVision, EMMA, Video-MME, LongVideoBench

*It took several days to conduct experiments, because we need to evaluate old models on new datasets and new models on old datasets. At the end, 11 of 108 old models are not tested on the new datasets, and those results will not be included in the following test.

The averaged RMSEs are shown as follows.

If we only know 20% performance of new models and new datasets

If we know 50% performance of new models and new datasets

Our method shows better generalization compared to the baselines. We observe that generalizing to new datasets is relatively easier than to new models. This is probably because new datasets are often very challenging, leading to generally lower model performance and smaller RMSEs. In contrast, new models often have novel designs and remarkable improvements. This makes generalization to unseen models more difficult and needs further exploration.

[1] Probabilistic matrix factorization. NeurIPS 2007.