Law of Vision Representation in MLLMs

Q3. Why does the AC score not exhibit strong performance on the TextVQA and VizWiz benchmarks? Are there any significant differences between these two benchmarks and other benchmarks?

As discussed in Section 6.3, benchmark performances are less correlated in OCR-based tasks, such as TextVQA and VizWiz, because the correspondence dataset measures the correspondence of natural images rather than text-heavy images. Consequently, the prediction of the optimal vision representation is less accurate for these benchmarks, requiring more finetuning runs to identify the optimal representation.

Q4. How is the number of fine-tuning in Tab. 1 determined? Why does the number of fine-tuning vary across different benchmarks?

The number of finetuning runs in Table 1 is determined to achieve an average Recall@3 of approximately $90$%, which we believe provides the best accuracy for presentation purposes. That being said, Table 1 is only a summary of the data shown in the paper Figure 4, where the full relationship between the number of fine-tuning runs and Recall@3 is detailed.

In practical use, the number of finetuning runs should follow the available computation budget, as explained in our response to Reviewer AFzz’s Weakness 1. The variation in the number of finetuning runs across different benchmarks is due to differences in Recall@3 performance across benchmarks, which is explained in more detail in our response to Question 3.

W3. Providing the computational cost and parameter counts for the different vision representations used and discussing how these factors might impact the AC scores

The table lists all the vision representations tested in this paper, ranked by FLOPs and parameter counts in descending order. From the results, we observe that neither the FLOPs nor the parameter count of the vision encoder directly correlates with the AC scores or model performance. Specifically, larger vision encoders do not necessarily result in better performance or higher AC scores, demonstrating that the size of the vision encoder is not the determining factor for cross-modal alignment or overall model effectiveness.

Questions:

Q1. How would using a stronger contrastive learning model, like BLIP, or a weaker model affect the results when calculating cross-modal alignment score?

We explore changing the reference feature used for A score calculation and report the fitting results across four vision benchmarks, both with and without the reference encoder included in the test set.

The first rows in our analysis correspond to the reported $R^2$ in the paper, demonstrating that when using CLIP, even if CLIP is excluded from the test set (no CLIP in fitting for the second row), the relationship between the AC score and performance remains as strong as when CLIP is in the test set.

We also tested another contrastive learning model, SigLIP (instead of BLIP, primarily due to time constraints), as the reference embedding for A score calculation. The results showed a lower fitting correlation coefficient compared to CLIP, both when SigLIP was included in the fitting set and when excluded. This finding justifies our choice of using CLIP as the reference embedding.

Q2. Does calculating the correspondence score require training an MLP to predict key points? If so, different vision representations may require different training hyper-parameters. Consequently, what approach should be taken to determine these optimal hyper-parameters? If the representation dimensions of the vision representation are not the same, then how can they be fairly compared?

No, we predict key points using an unsupervised method, so no hyperparameters need to be tuned. All vision representations calculate the correspondence score using the same hyperparameters, ensuring fair comparisons across different representations.

For more details on the correspondence score calculation, please refer to the pseudocode provided in the updated Appendix Section 6.

Thanks for your time in reviewing our work! Here, we address your concerns as following:

Weaknesses:

W1. Cross-modal alignment score is the maximum cosine similarity between each pair of embedding vectors from the CLIP embedding and the target embedding. However, there may be bias in CLIP characterization. At the same time, CLIP is also one of the vision representations involved in the test, which results in the CLIP representation having an artificially high score on the cross-modal alignment score.

The reference model for the A score is designed to be flexible and can be adjusted according to individual preferences. For example, one can choose a different model as the reference based on the belief that it has the highest cross-modal alignment or to minimize fitting error. So far, we believe that CLIP has the best cross-modal alignment, and it is therefore expected to have the highest score. We have also tested SigLIP as the reference embedding, as detailed in our response to Question 1, and found that the results still align with the proposed framework. Importantly, we are the first to propose and quantify the relationship between cross-modal alignment and MLLM performance, which we believe is already a significant contribution. We encourage future research to build upon our findings and further improve methods for quantifying cross-modal alignment.

W2. Verify the generalization of the proposed method on more MLLM structures or the base LLMs

In this paper, we demonstrate the relationship between the AC score and performance by fitting a regression model and reporting the $R^2$ value. As shown in paper Table 3, the $R^2$ achieves $95.72$% when averaged across four vision benchmarks under the LLaVA1.5 setting, where the LLM is Vicuna-7B-1.5.

In additional experiments, we show that the fitting $R^2$ remains strong, and in some cases even higher, when using different LLM types and sizes. The variation in $R^2$ falls within a reasonable range, indicating that the effect of LLM and vision representation compatibility, if it exists, is negligible compared to the influence of the A and C factors. These results demonstrate that the Law of Vision Representation relationship holds across different LLMs.

We strongly believe that the Law of Vision Representation applies to a broader range of LLMs. This is because the vision encoder serves as the initial feature extractor for vision information, and changes in the alignment module or LLM do not alter the quality of the information extracted. While the empirical validation of the Law could extend infinitely, finetuning and inference MLLMs is expensive, with the settings tested in this paper and rebuttal costing nearly $100,000. This underscores the significance of the Law of Vision Representation, as it enables others to avoid these high costs. We hope that the additional experiments provided convey the broad applicability of our findings.

Dear Reviewer 9bVh,

Thank you once again for the time and effort you’ve dedicated to reviewing our work. We have carefully replied to your weaknesses and questions you highlighted in your feedback and provided additional experiments as needed.

We kindly ask if you could let us know whether our responses have sufficiently addressed your concerns. Your continued feedback would be invaluable in helping us further refine the paper during the discussion phase.

We wish to kindly emphasize that some of the statements of Reviewer 9bVh may reflect a misunderstanding of our work:

1. Comparison between SigLIP and CLIP - It is never our intention to argue whether SigLIP or CLIP is a better model in terms of performance. In our work, we demonstrate that cross-alignment and correspondence of vision representations are positively correlated to the performance of MLLM. Therefore, we consider a diverse range of vision representations that could be used for MLLMs, regardless of their sizes. SigLIP and CLIP are merely data points that follow the law we discovered. This was added in Table 5 and 6 captions in the revised Appendix.

2. Generalization of AC Score - Our results show that the AC score generalizes to the combination of SigLIP ViT-SO400M/14@384 + DINOv2 ViT-L/14@518 + OpenAI CLIP ViT-L/14@336 + ConvNeXt-XXL@1024. According to the configuration outlined by reviewer 9bVh, which is a recognized current SOTA work, the Cambrian's [1] setting, we have also referenced it in our paper. To address reviewer 9bVh's concerns more comprehensively, we take the official checkpoints from the authors of Cambrian and evaluated the AC score. Additionally, we associated the AC score with the evaluation on MMBench, MME, and MMMU. The results presented in Cambrian's paper (Table 3) is consistent with our AC score. As shown in the below table, the AC score successfully indicate the lowest and highest performing vision representations, with bold indicate the highest for each benchmark.

3. Discussion on A and C Scores for CLIP@224 and CLIP@336 -

The above table compares A and C scores between CLIP@224 and CLIP@336, showing that CLIP@224 has a significantly lower C score, while CLIP@336 experiences a slight drop in A score. Since CLIP has a high A score, and A scores are expected to exhibit diminishing returns as they approach the higher range, the slight drop is anticipated to have minimal effect on predicted performance.

• CLIP@336’s higher C score than CLIP@224: This is reasonable as higher resolution better preserves visual details, leading to higher correspondence.

• CLIP@224’s higher A score than CLIP@336: To the best of our knowledge, no prior work has measured the cross-modal alignment of the same contrastive learning model with different input resolutions. Consequently, it is challenging to conclusively interpret the true cross-modality behavior in this case.

We have added this discussion to revised Appendix 4.

We wish to emphasize again that our work represents the first attempt to quantify cross-modality alignment in vision representations for MLLMs. Additionally, A score is intentionally designed to be flexible, and we encourage future works to refine it to eliminate potential biases. We believe our work has already made a significant contribution by proposing a meaningful correlation, which holds practical value for the future development of MLLMs.

[1] Cambrian-1:A Fully Open, Vision-Centric Exploration of Multimodal LLMs.

Thanks for your time in reviewing our work! Here, we address your concerns as following:

Weaknesses:

W1. The calculation of the A score by using the image feature of CLIP might not fully fulfill the motivation for quantifying the cross-modal alignment, as it assumes that the image-text features are perfectly aligned in CLIP.

While this limitation is acknowledged in the limitation section of our work, we want to note that the AC Policy is designed to establish only the relative order of vision representations within the search space. This means that we only need to assume that CLIP has the highest cross-modal alignment compared to other vision encoders, not "perfect alignment", for the policy to work. Moreover, the reference model for the A score is designed to be flexible and can be adjusted based on individual preferences, such as the belief that another model has the highest cross-modal alignment or to minimize fitting error.

Questions:

Q1. Please clarify the meaning and origin of the dimension 6 in $k^\prime$ in Policy Fitting. Is it related to the coefficients of the AC score?

Yes, the dimension 6 corresponds to the coefficients for the polynomial-transformed A and C scores. Without any transformation, fitting the A and C scores for each vision representation would involve two terms: $[A, C]$, in a linear regression. However, the AC score applies a second-degree polynomial transformation to the A and C scores, resulting in a six-term vector for fitting. In vectorized form, this is represented as $[1, A, C, A^2, A \cdot C, C^2]$.

Q2. Provide a pseudo code for the AC POLICY algorithm to enhance understanding.

Thank you for the suggestion. We have included the calculations for the A and C scores, as well as the AC Policy, in Appendix Section 6.

Q3. Explain the choice of Recall@3 as the main metric and supplement with other metrics (like Recall@1, Recall@2, etc.).

In Table 1 of the paper, we demonstrate that, on average, only $3.88$ finetuning runs are required to achieve $89.69$% Recall@3 in identifying the optimal vision representation. This means that, with a computation budget of at most $7$ runs, we can find the optimal vision representation in almost $90$% of the time. Using this same budget, we further evaluate other metrics, such as Recall@1 and Recall@2, to provide a comprehensive assessment.

We observe that, given the same computation budget where only 7 of vision representations in the search space can be tested, using Recall@1 provides a significant $20$% improvement over random trials in the chance of finding the optimal vision representation. However, using Recall@3 as a metric further optimizes this chance, demonstrating its practicality. This observation implies that, rather than assuming an absolute "optimal vision representation", we should acknowledge that each finetuning and inference process is subject to fluctuations, making "optimal" predictions inherently challenging. Employing Recall@3 accounts for these fluctuations, providing a more robust and practical approach without increasing computation costs.

Additionally, we provide the average performance difference between the Top-1 and Top-2 vision representations trained MLLM to illustrate that, in most cases, the performance differences are minimal and their order can be attributed to fluctuations:

Q4. Present the specific format (contains the weights and bias) of Policy Fitting and analyze the differences in fitting results across benchmarks.

Common behavior observed across most benchmarks:

• Positive coefficient for $A$ and $C$ terms: As A and C scores increase, benchmark performance increases proportionally to the coefficients of these terms.
• Negative coefficient for $A^2$ and $C^2$ terms: This indicates a diminishing return or saturation effect—while increasing A and C initially improves performance, their positive impact decreases as the scores grow larger.
• Positive $A$ and $C$ interaction term ($A \cdot C$): A positive coefficient for the interaction term implies that simultaneous increases in A and C magnify their combined effect on performance, resulting in a synergistic boost.

Comparing across benchmarks, we observe that for vision-based benchmarks (MMBench, MME, OKVQA, SEED-Bench), the coefficients for A and C terms are consistently positive, indicating that cross-modal alignment and correspondence are positively correlated with performance. They all have a heavier weight on A than C. However, we also note that the effect diminishes as A and C scores continue to increase, eventually plateauing (at the maximum score for the benchmark). This is due to the negative coefficients for the square terms, which represent a saturation effect where performance gains slow down as A and C scores grow.

For OCR-based benchmarks, such as VizWiz and ScienceQA, we observe unexpected coefficients. We attribute this to the issue discussed in Section 6.3. Specifically, our correspondence calculation image set is based on natural images rather than images containing text, which leads to inaccuracies in measuring correspondence for these tasks. As a result, the fitted function is less effective for OCR-based benchmarks.

Q5. SigLIP is supposed to have better cross-modal alignment as an improved version of CLIP, but it shows a lower score in your setting. Please explain.

SigLIP is a widely adopted vision encoder for MLLMs and has been shown to outperform CLIP on certain benchmarks under specific training data settings. However, this improved performance is not guaranteed to generalize across all settings. In our specific setting, as shown in Appendix 3, SigLIP demonstrates lower performance, which aligns with its lower A score measurement.

More importantly, to the best of our knowledge, no existing work has directly measured the cross-modal alignment between CLIP and SigLIP, and the original SigLIP paper does not mention any explicit design improvements for cross-modal alignment. As Reviewer 9bVh believes that BLIP is a stronger contrastive learning model than CLIP, this shows that cross-modal alignment remains an active area of research, and researchers have different beliefs on cross-modal alignment of different vision representations. Notably, our A score calculation framework is the first attempt in the MLLM field to quantify this factor.

Additionally, we conduct experiment using SigLIP as the reference embedding for the A score. If SigLIP is supposed to have better cross-modal alignment than CLIP, it would result in more accurate A scores and yield similar or higher fitting correlation coefficients. However, we observed a $3.73$% drop in the fitting $R^2$, indicating that SigLIP may be less reliable for measuring cross-modal alignment, at least under the current framework for quantifying it. This result suggests that SigLIP’s cross-modal alignment is questionable in this context. More detail, please see our response to Reviewer 9bVh’s Question 1.

Q6. Does a higher AC score always mean better performance?

Yes, the AC score and performance exhibit a linear relationship, meaning that as AC score increases, performance improves proportionally. Furthermore, we test various alternative relationships between the AC score and performance, confirming that the current conclusion is indeed the most accurate. For more details, please refer to our response to Reviewer JSYS’s Weakness 2.

Q7. Besides the A and C scores, are there other factors that need to be considered?

So far, we have no evidence suggesting that other factors need to be considered. For instance, we tested the effect of changing the LLM and found that the correlation between AC scores and model performance still holds. This indicates that the choice of LLM is not a significant factor. Please see our response to Reviewer JSYS’s Weakness 1 for more details.

Additionally, we investigate the relationship between vision encoder parameters and performance in our response to Reviewer 9bVh’s Weakness 3 and found no correlation. However, we encourage the community to explore additional factors that could build upon our findings, whether specific to certain benchmarks or applicable more broadly.

Dear Reviewer Nwyo,

Thank you once again for the time and effort you’ve dedicated to reviewing our work. We have carefully replied to your seven questions you highlighted in your feedback and provided additional experiments as needed.

We kindly ask if you could let us know whether our responses have sufficiently addressed your concerns. Your continued feedback would be invaluable in helping us further refine the paper during the discussion phase.

Q2. In Table 1, could you provide the top-1/2 predictions for a more comprehensive analysis?

In the paper Table 1, we demonstrate that, on average, only $3.88$ finetuning runs are required to achieve $89.69$% Recall@3 in identifying the optimal vision representation. This means that, with a computation budget of at most $7$ runs, we can find the optimal vision representation almost $90$% of the time. Using this same budget, we further evaluate other metrics, such as Recall@1 and Recall@2, to provide a comprehensive assessment.

We observe that, given the same computation budget where only $7$ of vision representations in the search space can be tested, using Recall@1 provides a significant $20$% improvement over random trials in the chance of finding the optimal vision representation. However, using Recall@3 as a metric further optimizes this chance, demonstrating its practicality. This observation implies that, rather than assuming an absolute "optimal vision representation", we should acknowledge that each finetuning and inference process is subject to fluctuations, making "optimal'" predictions inherently challenging. Employing Recall@3 accounts for these fluctuations, providing a more robust and practical approach without increasing computation costs.

Additionally, we provide the average performance difference between the Top-1 and Top-2 vision representations trained MLLM to illustrate that, in most cases, the performance differences are minimal and their order can be attributed to fluctuations:

Q3. Why does the 13 setting only require 12 finetuning runs to predict the optimal vision representations in a random setting?

First of all, in a random setting, we need to run all to find the optimal vision representation, but running all 13 out of 13 settings is no longer a prediction task. To maintain a fair comparison under the prediction problem framework, we use a probabilistic baseline. Specifically, we set the acceptable prediction Recall@3 at approximately $90$%. To achieve this level of performance in the random setting, 12 finetuning runs are required.

W3. It is confusing that “if the vision representation ensures accurate correspondence, more visual details are considered even if not attended by the text token”. This statement implies that a high correspondence vision representation preserves visual details, which is ungrounded.

We would like to point out that it is grounded that accurate correspondences in vision representation mean that more visual details are considered. Finding correspondences between images has been a core topic in computer vision for decades. It focuses on identifying semantically and geometrically similar points across different images given any keypoints. This goal aligns closely with the purpose of learning visual representations that many researchers in computer vision aim to achieve. Back to classic methods like SIFT [1] to modern powerful visual encoders like DINOV2 [2], researchers have worked to create representations that are robust to changes in appearance, visual illumination, local deformation, and viewpoints, all of which aligns the goal of finding correspondences. To this end, checking correspondences is now a key method to evaluate the ability of understanding visual cues in the representations, as seen in works like [3, 4]. It is also used as an objective for representation learning, as demonstrated by methods like [4, 5].

[1] Distinctive Image Features from Scale-Invariant Keypoints. IJCV 2004

[2] DINOV2: Learning Robust Visual Features without Supervision.

[3] Probing the 3D Awareness of Visual Foundation Models. CVPR 2024

[4] 3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features. CVPR 2024

[4] Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles. ECCV 2016

[5] SuperPoint: Self-Supervised Interest Point Detection and Description. CVPR 2018

Questions:

Q1. In section 3.2, the explanation of key points and their prediction process is missing.

We have added key points prediction process as part of the C score calculation, included as pseudocode Appendix Section 6. The key points in correspondence finding refer to distinctive points or regions in an image that can be identified and matched across different images. For visualizations of key points, refer to [1]’s website. In our work, we directly use the key points provided by the SPair-71K dataset and compute the correspondences based on them.

[1] Telling Left from Right: Identifying Geometry-Aware Semantic Correspondence. CVPR 2024

Thanks for your time in reviewing our work! Here, we address your concerns as following:

Weaknesses:

W1-1. How should we choose the number of $k^\prime$ for Policy fitting?

We should choose the number of $k^\prime$ based on the available computation budget. If one has the time and budget to test $10$ vision representations, they should test $10$; if they are willing to finetune for $100$ runs, they should do $100$. Ideally, we would hope to test every single vision representation in existence, which would guarantee finding the optimal vision representation for MLLM. However, the vast number of representations and the complexity of feature combinations make the search space incredibly large. In the real world, everyone tests only a subset of vision representations for MLLM, sacrificing chances of finding the optimal representation. This is why our work stands on a practical assumption: one should finetune the number of MLLMs based on their maximum budget, and AC Policy is designed to maximize the chance of finding an optimal vision representation for their MLLM.

W1-2. Given the outliers in the number of finetuning across benchmarks, on what kind of benchmark should we allocate the more computation budget for AC Policy?

Based on our answer to question 1, we would like to address your concern about the outliers in the number of $k^\prime$ by approaching it in the context of this question. I suggest allocating more finetuning runs if the task evaluation set differs significantly from the correspondence score calculation set. For example, if the benchmark is testing the OCR ability of an MLLM while the correspondence dataset evaluates correspondence on natural images, the performance correlation would be lower. As discussed in Section 6.3, benchmark performances are less correlated in OCR-based tasks because the correspondence dataset measures the correspondence of natural images. Consequently, the prediction of the optimal vision representation would be less accurate, thus requiring more runs to find the optimal representation. To make the AC Policy more practical, we not only provided an explanation for the outlier but also suggest that any future contributions include correspondence evaluation sets specifically for images containing text.

W2. All the experiments are conducted under the condition of freezing the Vision Encoders. The conclusions may change when the Vision Encoders are unfrozen, which limits the generalizability of the AC Policy and these conclusions.

To ensure scientific rigor in framing the Law of Vision Representation in MLLM, we clearly state the assumptions and conditions under which it holds. We emphasize that unfrozen vision encoders fall outside the scope of the Law of VR, as this scenario breaks both our assumptions and the framing of the question in two key ways:

1. Theoretical Aspect: We assume that the vision representation is the only independent variable, while the alignment module and LLM architecture remain fixed (line 145). However, with an unfrozen vision encoder, we cannot guarantee that the vision encoder does not take the function of the alignment module. This causes the architecture and role of the alignment module to change alongside the encoder, making the experiment uncontrolled and the models no longer comparable.

2. Application Aspect: A key application of the law is to determine the optimal vision representation before training the MLLM. With an unfrozen vision encoder, the vision representation evolves during stage 2 [1] or even up to the final stage [2, 3]. This makes it impossible to identify the optimal vision representation without completing the full training process. While it would be interesting to explore the scenario of unfrozen encoders, this case and exploration lacks practical use because people need to train until the final stage anyways.

Therefore, our conclusions are framed specifically within the context of frozen vision encoders, and the AC Policy is designed to operate under these assumptions.

[1] Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond.

[2] LLaVA-OneVision: Easy Visual Task Transfer.

[3] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs.

Dear Reviewer AFzz,

Thank you once again for the time and effort you’ve dedicated to reviewing our work. We have carefully replied to the three weaknesses and three questions you highlighted in your feedback and provided additional experiments as needed.

We kindly ask if you could let us know whether our responses have sufficiently addressed your concerns. Your continued feedback would be invaluable in helping us further refine the paper during the discussion phase.

W3. Although the authors provide a theoretical explanation of the impact of the AC score on performance, the paper would be strengthened by a more rigorous proof or formal derivation of the mathematical relationship. Additionally, a more in-depth discussion on the design and application of AC scores, encompassing both explainable and theoretical aspects, should be included.

We want to emphasize that ours is the first work in the MLLM field to attribute model performance to specific network design characteristics. Empirically, we demonstrated that the AC score is linearly correlated with model performance, achieving a strong correlation with an $R^2$ of $95.72$. In addition, we provide a theoretical explanation that the AC score is positively correlated with MLLM performance. Influential works such as the Scaling Laws [1, 2] also adopt an empirical approach, demonstrating relationships without formal proofs. Therefore, while we recognize the value of a formal derivation, we are confident that our findings provide valuable insights, even in the absence of a formal mathematical derivation.

[1] Scaling Laws for Neural Language Models.

[2] Training Compute-Optimal Large Language Models.

Questions:

Q2. The experimental setting for the chosen LLM and projector is unclear. Could you give more information about the experimental setting?

We follow the experimental settings used in LLaVA 1.5, which is one of the most widely adopted and generalizable MLLM architectures. Specifically, the base LLM is Vicuna-7B, and the projector consists of two linear layers with a GeLU activation in between.

Thanks for your time in reviewing our work! Here, we address your concerns as following:

Weakness:

W1. Additional experiments should be carried out using various LLM sizes and projector designs to showcase wider applicability.

In this paper, we demonstrate the relationship between the AC score and performance by fitting a regression model and reporting the $R^2$ value. As shown in Table 3 of the paper, the $R^2$ achieves a strong $95.72$ when averaged across four vision benchmarks under the LLaVA1.5 setting, where the LLM is Vicuna-7B-1.5.

In the above additional experiments, we show that the fitting $R^2$ remains strong, and in some cases even higher, when using different LLM types and sizes. The variation in $R^2$ falls within a reasonable range, indicating that the effect of LLM and vision representation compatibility, if it exists, is negligible compared to the influence of the A and C factors. These results demonstrate that the Law of Vision Representation relationship holds across different LLMs.

We strongly believe that the Law of Vision Representation applies to a broader range of LLMs and projectors. This is because the vision encoder serves as the initial feature extractor for vision information, and changes in the alignment module or LLM do not alter the quality of the information extracted. While the empirical validation of the Law could extend infinitely, finetuning and inference MLLMs is expensive, with the settings tested in this paper and rebuttal costing nearly $100,000. This underscores the significance of the Law of Vision Representation, as it enables others to avoid these high costs. We hope that the additional experiments provided convey the broad applicability of our findings.

W2. A justification for the linear assumption or a comparison with other regression models would strengthen the analysis.

In our work, we use a linear regression model to capture the relationship between the AC score and model performance. To clarify, the AC score is a second-degree polynomial transformation of the A and C scores. This means that we are effectively applying polynomial regression through the transformation of A and C before fitting a linear regression model to predict performance.

As shown in the paper Table 3, we compare the results of using no transformation (simple linear regression) and polynomial transformation (polynomial transformation + linear regression = polynomial regression), demonstrating that the latter provides better predictive performance. To strengthen our analysis, we conducted additional experiments using other regression models. Here, we calculated the train and validation mean square error (MSE) using a leave-one-out cross-validation (LOOCV) strategy on 13 vision representations, averaged across 4 vision-based benchmarks. These results further validate the effectiveness of our approach.

By observing and trading off between train and validation MSE, we find that our method best captures the relationship between the AC score and model performance. Simple linear regression without polynomial transformation shows underfitting as shown in high train MSE. In contrast, higher-degree polynomial regression on A and C leads to overfitting as shown in high validation MSE. Logarithmic regression, on the other hand, exhibits high train MSE, indicating that it does not accurately model the relationship. These findings support our choice of using a second-degree polynomial transformation combined with linear regression as the most appropriate approach.

Additionally, the use of linear regression on top of the polynomial transformation of the A and C scores offers excellent explainability by allowing direct inspection of the weights and biases. For more details, please refer to our response to Reviewer Nwyo’s Question 4. In summary, we selected our current modeling technique based on its strong empirical performance and superior explainability.

Dear Reviewer JSYS,

Thank you once again for the time and effort you’ve dedicated to reviewing our work. We have carefully replied to the three weaknesses you highlighted in your feedback and provided additional experiments as needed.

We kindly ask if you could let us know whether our responses have sufficiently addressed your concerns. Your continued feedback would be invaluable in helping us further refine the paper during the discussion phase.