[1] The motivation is unclear. In the introduction (L081-085), the author states two challenges, i.e., the strategy of implicit transferability modeling remains largely unexplored and the implicit modeling process requires the final embedding states of the model after fine-tuning. However, it can not fully understand the where the challenge is. The author more specifically explain the challenges.

[2] The description of the method is not clear. For example. In L221-L222, the embedding space division: The author is suggested to briefly introduce the motivation of division, what to divide and how to divide. Currently, I can not fully understand the motivation and implementations.

[3] The pseudo-clustering accuracy is essential to transferability estimation. The authors are suggested to evaluate the sensitivity of the clustering performance.

1. In line 225, the methodology mentions "K independent subspaces sampled from E"; however, the process for obtaining these divisions is not adequately described. The criteria for determining K and the methodology used to derive these independent subspaces require further clarification. It would be helpful if the authors could provide specific examples of how K is determined and the practical sampling process for the subspaces.

2. In line 307, the generation of pseudo-cluster centers is noted to involve various methods, including the use of random vectors from high-dimensional space. This approach appears to conflict with conventional methods (centers are determined by some clustering methods) but lacks sufficient explanation within the text.

3. The evaluation of the proposed method is conducted solely on a single benchmark, omitting more widely recognized benchmarks (used in SFDA[1], Logme[2]), which limits the robustness of the findings.

[1] Not All Models Are Equal: Predicting Model Transferability in a Self-challenging Fisher Space.

[2] LogME: Practical Assessment of Pre-trained Models for Transfer Learning

Major weaknesses

1. The biggest weakness is that the method is not clearly presented, while conceptually the elements make sense, the interconnection and the final TE method (or scoring) is not directly clear from the manuscript. To make this weakness more concrete:

   • (major) The final transferability estimation score (s_i) for model i (for a particular target task) is not explicitly defined. It is related to the ideal mapping gamma, but not provided.
   • (major) The loss (eq 7) uses Ê (E-hat), which is defined above Eq 7 as the final embedding state after fine-tuning. Defining a transferability metric which works after fine-tuning is probably not the goal, so please clarify.
   • (major) How is the lambda parameter (Eq 8), controlled by the learning rate (eta) in Eq 7 as stated in L331. How is Eq 8 in general related to Eq 7? Eq 8 defines a joint loss, Eq 7 only an update of the parameters.
   • (major) What is the objective driven loss? How is it defined?
   • (major) In the implementation details it is stated that some of the target dataset is used for validation purposes in ITM. What parts are validated, and when in the pipeline, and how is that used?
   • The difference between mapping Gamma (L222) and mapping Phi (L224) is not clear to me. And what is the difference between psi(.,.)(L224) and psi(.) (L245)?
   • What is z, it seems only implicitly used in the (approximation) of ^Z (Z-hat)? Moreover, the decomposition of q_phi(z|E), is purely based on the independence assumption of E, I don’t think a double bayes rule is necessary to write that down. Finally, how is q used? Is the loss in Eq 4, a sum over all j in K (of E 3)?
   • What is the difference between the subsets A (indexed by j) and the subset of K (also indexed by j)?
   • What does it mean that the process is “treated as an interaction between the model’s transferability and downstream tasks, leading to more effective and adaptable estimation across a broader range of scenarios.” What interaction is used, is the model learned once for multiple downstream tasks? Does it then generalize to others?

2. The second biggest weakness of the current study is the experimental evaluation, although this is a more general problem within transferability estimation literature, but the current experimental evaluation is too weak to draw any conclusions. The main problem is that a single ranking experiment is evaluated, while the outcome of this ranking really does depend on the selected individual models in the model-zoo. This means that adding or removing a single model to/from the zoo all results are likely to be significantly different. This has been studied in [AgostinelliECCV22]. Therefore, the method should be evaluated on a larger set of ranking experiments. This is not too complicated nor too computational expensive: one could draw samples from a larger model-zoo. For example one could draw all sets of 10 models out of a model-zoo of 14 models, this provides (approx) 1000 ranking experiments (14 choose 10). This only requires computing once the scores (s) and ground truths (r) for 4 additional models, while providing 999 more ranking tests. Without such an experiment, I think it is impossible to draw any conclusion on the success of any method.

Minor weaknesses

• Figure 1: I’m unsure what I see (and should see) in this plot. It is remarkable that the MAE-B16/SimMIM-B16 models have different GT radar plots between (a) and (b).
• Figure 2: The y-scales are different from plot (a) and plot (b).
• (nitpick) In the related work the cite command is used, where it should be the citep, for example: NCE Tran et al. (2019) → NCE (Tran et al., 2019).
• (nitpick) The sub-index i is used for the number of elements in the dataset (L186) and for the number of models in the zoo (L187)
• It is unclear how many epochs over the target task training data are performed by each method. Most transferability estimation methods assume (eg) 1 epoch (to get feature embeddings and target labels).
• In the abstract and introduction ‘generalizability’ of a transfer estimation method is mentioned, what do you mean with that?

References

• [AgostinelliECCV22] Agostinelli et al, How stable are Transferability Metrics evaluations?, ECCV 2022.

1. In Lines 240-243 and 268-269, the authors state that 'the division and the recursive formulation reduce overall complexity.' It is recommended to substantiate this claim with both theoretical analysis and quantitative experiments.
2. What are the main contributions or innovations of the embedding space division and dynamic equation-based approach compared to existing methods?
3. For different generation methods in Pseudo-cluster center constraints, could the authors provide theoretical analysis alongside the empirical results?
4. It is recommended to include more comparison methods from recent 2024 publications.
5. A discussion on the limitations of the proposed approach would be valuable.
6. Thorough proofreading and refinement of illustrations would enhance the clarity and quality of the paper.

• Figures are not well-used in the paper.
  • Too many figures (i.e., Figure 1 and 2) are used to describe the well-known problems in the field of transferability estimation.
  • Figure 1 is hard to understand at the first glance, particularly when it includes many abbreviations that are explained in the later literature.
  • Too much contents are included in the Figure 3, making it difficult to read at the first glance.
• There are several confusions in the paper:
  • If Equation (7) is re-written from Equation (6) by denoting C, then a $W^n$ term is missing in the formula. I doubt the mathematical correctness of Equation (7) and the related discussion on the benefits of this recursive form.
  • In Section 3.4, it has been mentioned that $\lambda$ is a hyper-parameter and is controlled by C. However, the paper doesn't specify how this is achieved.
  • In Section 3.4, it has been mentioned that the embedding pre-standardization is applied for various benefits. However, it doesn't specify how this is achieved as well.
  • In Section 3.4, it has been mentioned that the iteration number in DCA is fixed to 1. However, it doesn't specify the purpose of this setting. Also, if there is only one iteration required, what would be the benefits of transforming the update of the embedding space into a recursive formulation as in Equation (7)?
  • In the Conclusion, it has been claimed that "We conduct experiments on recent models and a diverse set of downstream tasks,...". However, experiments are only conducted on the image classification task.