Unleashing the power of Neural Collapse for Transferability Estimation

We thank the reviewer for providing valuable comments and address your concerns in the following responses.

[Q1]: "On the role of neural collapse in transfer learning. [1]" ICLR 2022. The authors should discuss this paper and the contributions of the submission related to that.

[A1] Thank you for your kind reminder. We added a discussion in the related work section.

[1] investigates the impact of pre-trained networks undergoing neural collapse on source training data on various types of target datasets. The authors find that neural collapse occurring on the source data can generalize to new samples and even unseen categories. We list the main difference from [1] below:

• The task and setting are different. [1] study the ability of models, which undergo neural collapse on source tasks, to learn representations for classification that are transferable to new, unseen classes. However, our task is to propose a metric to estimate the transferability of a given pre-trained model. Whether this model undergoes neural collapse on the source data is unknown.
• [1] primarily investigates the performance of models experiencing neural collapse in experiments to learn about phenomena and draw conclusions. In contrast, our work is inspired by the characteristics of neural collapse to propose a method for transferability estimation.

[Q2]: The authors claim that a key component of their method is the class fairness term. It is unclear how FaCe compares to a variant without that term. There is no ablation to understand how this part affects the final metric.

[A2] We agree that the ablation study is crucial. As shown in Section 4.6, we provide the ablation study of two terms of FaCe. The results validate that both terms are effective, and they complement each other to achieve the best average performance.

[Q3]: The experimental validation is weak. A more extensive evaluation is needed to showcase any improvements of FaCe vs the other metrics.

[A3] We follow existing work and select a set of common-used representative target datasets. These datasets cover various data types, such as animals, traffic signs, textures, and more, providing a diverse set of evaluation scenarios.

In Appendix A.3, we add experiments on the large-scale dataset food101 (~101,000 images) in Table 1, and include more target datasets in Table 2, 3.

As shown in Appendix A.3, Table 6,7,8, in the presence of a large-scale dataset, we still achieve competitive average performance (Table 6). Moreover, on more target datasets, we also obtain the best average performance on the other two model zoos (Table 7, 8).

[Q4]: A very related concurrent work from ICCV 2023 should be cited and possibly also discussed (as concurrent work, of course, not limiting this paper's novelty):

[A4] Thank you for your kind reminder. We have also noticed the paper [2], which was released after our submission and shares similar ideas with ours. We have added a discussion about it in the related work section.

NCTI (their method) is a concurrent work of our method, which is also inspired by neural collapse. NCTI consists of three terms, which correspond to three characteristics of neural collapse, respectively. Different from NCTI, which assesses the geometry structure based on nuclear norm, FaCe is based on the class distribution overlapping matrix entropy. We will make effort to reproduce this work and include a comparative analysis with it in the future.

• [2] How Far Pre-trained Models Are from Neural Collapse on the Target Dataset Informs their Transferability. ICCV 2023.

[Q5]: What is only a subset of papers presented in Tab 2 and 3? Can you provide results for all, eg in an appendix if the issue is space?

[A5] In Table 1-3, the comparison methods are the same (LEEP, NCE, LogME, H-score, GBC).

We suspect that when you mentioned "a subset of papers," you might refer to the target datasets. We also provide more results in Appendix A.3. On these new target datasets, FaCe still achieves state-of-the-art performance.

Dear reviewer,

We genuinely appreciate the time and effort you've invested in reviewing our paper. We have carefully provided relevant responses and results to your concerns. We are eager to further discuss with you and gain your insights before the end of the Author/Reviewer phase. Please let us know if any aspect of our work remains unclear or if you have additional feedback.

Thank you.

Alright, thanks for clarifying an important part of the tuning for the task that I confused. My apologies again, it is busy times.

Based on the author responses I will increase my score to borderline accept.

We thank the reviewer for providing valuable comments and address your concerns in the following responses.

[Q1]: The first term of NC is common. This idea is commonly used in various classification tasks.

[A1] We agree that the first term of NC is commonly used in various classification tasks. However, our work introduces the first term of NC as a metric during the downstream fine-tuning process, as shown in Figure 1. Motivated by the observation that the NC of the pre-trained model is correlated with the corresponding fine-tuned model, we include the first term of NC, i.e., the variance collapse term, for transferability estimation, which could be considered as a baseline method of the proposed FaCe.

Even for this term, we want to clarify the difference between the variance collapse term and several closely related works, Hscore [1] considers between-class variance and feature redundancy; GBC [2] is based on the summation of the pairwise class separability. Our variance collapse term is the magnitude of between-class covariance compared to within-class covariance, which is different from them.

• [1] An information theoretic approach to transferability in task transfer learning
• [2] Transferability Estimation Using Bhattacharyya Class Separability

[Q2]: Class fairness score F is quite similar to making the distances between these distributions equal.

[A2] The high-level idea that makes the distances between class distributions equal is indeed used by some classification methods. However, such a perspective has not been explored in the transferability estimation task. Also, we do not directly borrow ideas from other works but design a specific metric for our task.

The main insight lies in the observation that the NC metrics are robust along the fine-tuning process, motivating us to estimate the degree of NC to estimate the fine-tuned accuracy, i.e., the transferability.

Different from the score in NC2, we propose a similar formulation that estimates equidistance based on the distribution overlap matrix entropy. We also provide a comparison with NC2 in Appendix A.2, the results validate that our method is more suitable for transferability estimation problem and achieve a higher performance.

[Q3]: The fine-tuning hyperparameters of different pre-trained models may have a significant impact, and the phenomena observed in the paper might lack persuasiveness.

[A3] We follow existing works [1,2,3] in transferability estimation and select a set of hyperparameters that perform best on the downstream dataset. In fact, the results are obtained based on different seeds, indicating the robustness of the fine-tuning process. We have clarified this point in the experimental details of Section 4.1.

• [1] Transferability Estimation using Bhattacharyya Class Separability.
• [2] LEEP: A New Measure to Evaluate Transferability of Learned Representations.
• [3] LogME: Practical Assessment of Pre-trained Models for Transfer Learning.

[Q4]: lacks experiments on the relationship between the accuracy of pre-trained models, the accuracy of fine-tuned models, and the proposed FaCe.

[A4] In revision, we updated the zip file of supplementary materials, which added a file named "Acc-FaCe.xlsx." This file provides the accuracy and FaCe values of our experiments. Besides, Figure 4 presents the qualitative results of the fine-tuned model accuracy and FaCe Score, along with comparisons to other methods. However, the classifier dimension of the pre-trained model does not match the number of classes in the downstream target data, making it impossible to provide accuracy directly for the pre-trained model.

[Q5]: The authors observed an interesting phenomenon, it may not be solid.

[A5] We agree. Our main insight is based on a preliminary study, motivating us to develop a new score FaCe. The two terms of FaCe correspond to two characteristics of NC. To validate the effectiveness, we conducted numerous experiments, and found it works quite well across various model zoos and tasks. We believe the proposed method FaCe would shed some light on future work.

[Q6]: They have not clarified the differences between their metric and those presented in other papers.

[A6] We feel sorry for any confusion. As written in Section 2, previous works focused on between-class separability and the information of features, which is similar to the high-level idea of our variance collapse term.

Compared to existing methods, we further consider the class fairness of pre-trained models towards the target class, this is a new perspective, which has not been explored by existing methods.

Dear reviewer,

We genuinely appreciate the time and effort you've invested in reviewing our paper. We have carefully provided relevant responses and results to your concerns. We are eager to further discuss with you and gain your insights before the end of the Author/Reviewer phase. Please let us know if any aspect of our work remains unclear or if you have additional feedback.

Thank you.

Dear Reviewer,

Since the discussion deadline is approaching in less than 11 hours, we kindly request your feedback on whether the response adequately addresses your concerns. If you have any more questions, we would be happy to provide further clarification.

Your timely response is greatly appreciated.

Thanks.

We thank the reviewer for providing valuable comments and address your concerns in the following responses.

[Q1]: Considering the rapid dissemination of models online by third-party sources, the approach of selecting the optimal model for fine-tuning might have constrained applicability.

[A1] It is precisely because there are numerous third-party models that it becomes necessary to evaluate these models to estimate their suitability for downstream tasks without time-consuming fine-tuning for all the models.

[Q2]: The assertion of a strong correlation is not clearly evident from the paper. For example, It would be beneficial if, from the experimental results, some takeaway messages were more explicitly provided.

[A2] We feel sorry for any confusion. We want to explain the strong correlation observed in Figure 1 as follows. The NC score ranking in these pre-trained models remains mostly consistent during fine-tuning. For instance, in subfigure (a), the yellow line represents the variation curve in NC for ResNet152 during the fine-tuning process. It starts with the highest NC on the target data from the initial moment and continues to maintain the highest NC throughout fine-tuning. The black curve (DenseNet201) starts with the lowest degree of NC, and continues to maintain the lowest level during the fine-tuning process. Other models also exhibit similar patterns, therefore, we posit the existence of a strong correlation between the NC of the pre-trained model and that of the fine-tuned model.

[Q3]: In its current shape the paper requires visiting the experimental result section and understanding which specific model or architecture best transfers according to the proposed metric.

[A3] Our metric does not exhibit bias towards specific network structures or loss functions. For example, on the DTD dataset, FaCe gives ResNet152 the highest score, while on Cifar-10, FaCe gives EfficientNetB3 the highest score.

This sounds reasonable for a qualified transferability estimation score since the best model is related to the specific downstream task. For different tasks, the best model, loss function or source datasets vary.

The task of transferability estimation aims to find a metric for assessing the appropriateness of pre-trained models for downstream tasks, rather than evaluating the effectiveness of particular architectures or loss functions.

[Q4]: The paper frequently refers to the "domain shift" causing features not to lie on the hypersphere. The implications of this assertion are not clear, and what is meant by it being "too strict," needs clearer elaboration. The paper seems to lack a direct discussion on this.

[A4] When Neural Collapse (NC) occurs, the features of the training data collapse to class means, and the class means, along with the classifier weights, converge to the vertices of a simplex equiangular tight frame (ETF). At this point, the class means of the features can be considered as distributed on a hypersphere [1]. However, the model is pre-trained on the source data. On downstream target data, domain shift exists, and the class means will not be located on the hypersphere.

Our class fairness term is inspired by NC2. A widely-used metric in existing literatures is defined as $\mathcal{NC}_2(W) = \left\|\frac{W W^{\top}}{\left\|{W} {W}^{\top}\right\|_F}-\frac{1}{\sqrt{K-1}}\left({I}_K-\frac{1}{K} \mathbf{1}_K \mathbf{1}_K^{\top}\right)\right\|_F.$ Essentially, this metric measures the equiangularity of the classifier weights or class means. To use this metric, it should be under the scenarios where the class means are located on the hypersphere. This is the meaning of "too strict" in our manuscript.

We apologize for any confusion caused by the unclear description. We have clarified the relevant description in Section 3.2.

• [1] Neural Collapse: A Review on Modelling Principles and Generalization. TMLR 2023.

[Q5]: Could the final classifier bias parameters be disregarded, thereby naturally aligning features with the hypersphere?

[A5] The classifier is pre-trained on the source dataset, which has a different label space from the downstream target dataset, therefore, using the classifier makes no sense.

[Q6]: The metric introduced appears to hold even when training does not proceed until TPT. Given this, what is the significance of emphasizing the necessity for neural collapse in the proposed method?

[A6] Sorry for the misunderstanding. We do not assume Neural Collapse in fine-tuning the pre-trained model. By contrast, we study the degree of NC of pre-trained models during fine-tuning, through the process and find an interesting observation. Our method is motivated by this observation, to estimate the degree of NC of the pre-trained model, and it is independent of whether training has reached TPT.

Dear reviewer,

We genuinely appreciate the time and effort you've invested in reviewing our paper. We have carefully provided relevant responses and results to your concerns. We are eager to further discuss with you and gain your insights before the end of the Author/Reviewer phase. Please let us know if any aspect of our work remains unclear or if you have additional feedback.

Thank you.

Dear Reviewer,

Since the discussion deadline is approaching in less than 11 hours, we kindly request your feedback on whether the response adequately addresses your concerns. If you have any more questions, we would be happy to provide further clarification.

Your timely response is greatly appreciated.

Thanks.

We thank the reviewer for providing valuable comments and address your concerns in the following responses.

[Q1]: The introduction contains repetitive statements regarding transferability estimation and its objectives. For clarity and brevity, such redundancy should be avoided.

[A1] Thanks for your suggestion, we have updated the related description in the Introduction.

[Q2]: The meaning of the variable 't' in Equation (6) is not defined in the surrounding context.

[A2] Sorry for the typos, the variable t is the temperature of softmax, we have updated the related description in Section 3.2.

[Q3]: In Variance Collapse calculation, equal weight is given to all categories, yet performance evaluation gives more importance to larger sample size categories. This setup may affect the representation of FaCe's performance on the target dataset.

[A3] The final fine-tuned accuracy is computed on the test set. In practice, we may not know the class distribution of the test set when calculating the score, so it's not possible to pre-assign weights to certain classes. What we can obtain is the class distribution of the training set. In downstream tasks, to ensure generalization on the test set, the class distribution of the training set is often balanced.

[Q4]: "equal distance of each class to the other classes" is not synonymous with "equal distances amongst all classes." The claim "Due to the domain shift, the features do not lie on the unit sphere" lacks sufficient justification.

[A4] When Neural Collapse (NC) occurs, the features of the training data collapse to class means, and the class means, along with the classifier weights, converge to the vertices of a simplex equiangular tight frame (ETF). At this point, the class means of the features can be considered as distributed on a hypersphere [1]. However, the model is pre-trained on the source data. On downstream target data, domain shift exists, and the class means will not be located on the hypersphere.

Our class fairness term is inspired by NC2. A widely-used metric in existing literatures is defined as $\mathcal{NC}_2(W) = \left\|\frac{W W^{\top}}{\left\|{W} {W}^{\top}\right\|_F}-\frac{1}{\sqrt{K-1}}\left({I}_K-\frac{1}{K} \mathbf{1}_K \mathbf{1}_K^{\top}\right)\right\|_F.$ Essentially, this metric measures the equiangularity of the classifier weights or class means. To use this metric, it should be under the scenarios where the class means are located on the hypersphere.

Hence, we chose not to use the equiangularity measure of NC2 as found in existing work but instead propose to approximate NC2 using the equidistance of each class distribution.

We apologize for any confusion caused by the unclear description. We have clarified the relevant description in Section 3.2.

• [1] Neural Collapse: A Review on Modelling Principles and Generalization. TMLR 2023.

[Q5]: The experimental section lacks ablation studies on Class Fairness. To substantiate the effectiveness of the method, the inclusion of relevant experimental evidence is imperative.

[A5] We agree that the ablation study is crucial. As shown in Section 4.6, we provide the ablation study of two terms of FaCe. The results validate that both terms are effective, and they complement each other to achieve the best average performance.

[Q6]: While employing FaCe as the criterion to judge the generalizability of pre-trained models in target domains, have the authors considered the impact of differences in class distributions and class counts between source and target domains?

[A6] In transferability estimation, only the pre-trained source model is available, and the information about the source data is unknown. We believe that the only source model available is a more practical setup. In this setting, we cannot obtain the class distribution of the source or consider differences between the source and target. Your idea is reasonable, and we will consider about it in future work.

[Q7]: The authors extended the concept of equiangularity to equidistance. In the process of calculating distances between classes, why was the current method chosen? Were alternative approaches contemplated?

[A7] The most naive approach is to represent a class using its class mean and compute the class mean (CM) distances. However, due to significant differences in the intra-class variances of different classes in the target data, it is more reasonable to fit Gaussian distributions to the feature distributions of different classes and calculate the overlap. When measuring overlap between distributions, the introduction of Bhattacharyya (Bh) distance is a common practice.

We provide experiments using class mean distances as an alternative to measuring class distribution overlap below. Table (a) (b) and $c$ correspond to Tables 1 2 and 3, in our paper, respectively (i.e., (a) heterogeneous model zoo with a single source, (b) heterogeneous model zoo with multiple sources, and $c$ homogeneous model zoo with multiple sources and loss functions). The values in the table are the average results. It can be seen that FaCe with Bhattacharyya distance (FaCe W/ Bh) has a better average performance.

Dear reviewer,

We genuinely appreciate the time and effort you've invested in reviewing our paper. We have carefully provided relevant responses and results to your concerns. We are eager to further discuss with you and gain your insights before the end of the Author/Reviewer phase. Please let us know if any aspect of our work remains unclear or if you have additional feedback.

Thank you.

Dear Reviewer,

Since the discussion deadline is approaching in less than 11 hours, we kindly request your feedback on whether the response adequately addresses your concerns. If you have any more questions, we would be happy to provide further clarification.

Your timely response is greatly appreciated.

Thanks.

[Q4]: How about replacing Variance Collapse with inverse Bhattacharyya coefficient for simplicity of formulation?

[A4] The Bhattacharyya coefficient measures the overlap between class distributions and quantifies the separation between them. The intuitive difference between the Bhattacharyya coefficient and Variance Collapse score is that VC additionally considers the compactness within classes.

Your idea is reasonable as well. We provide the comparisons of these two methods on the heterogeneous model zoo with a single source (Table 1 in our paper) below. FaCe w/ VC is the original version of FaCe, and FaCe w/ neg-Bh is a variant that replaces our variance collapse term with the sum of the negative Bhattacharyya coefficient between classes. Specifically, we replace the score $C$ in FaCe with: $C_{negBh}=- \sum_{i=1}^{K}\sum_{j=1，i\neq j}^{K} B(k_i,k_j).$ The results are shown below. Our ultimate goal is to consider models that exhibit both large inter-class distances and relatively uniform class distributions as better. While replacing the first term with other forms may have some impact on the final results, the effect is not significant.

We thank the reviewer for providing valuable comments and address your concerns in the following responses.

[Q1]: Between-class covariance, $\Sigma_B$, is defined as the distance between the class avg and the global avg, which does not align with the definition of between-class, a distance between classes.

[A1] Here, we directly adopt the common definition of variance. Intuitively, the distance between the class avg and the global avg in $\Sigma_B$ serves the purpose of quantifying the total distance between all class means and the global mean. When the distances between classes are large, the class means will be farther away from the global mean, and when the distances are small, they will be closer to the global mean. Therefore, $\Sigma_B$ can effectively reflect the distance between classes.

[Q2]: Class Fairness term might be related to the Variance Collapse term. The Class Fairness term uses intra-class and inter-class covariance, which are also used in the Variance Collapse term. However, the relationship between the two terms is not studied enough.

[A2] Variance Collapse score (VC) is the magnitude of between-class covariance compared to within-class covariance. Different from that, the covariance in Class Fairness score (CF) is used to estimate the parameters of Gaussian distributions when modeling class features. These two terms complement each other, and we have conducted ablation experiments in Table 4 to investigate their individual effects.

Thanks for your suggestion, we have included some explanations of the relationship between the two terms in Section 4.6.

[Q3]: Why does between-class covariance $\Sigma_B$ in Variance Collapse use global average $h_G$ instead of class average $h_k$? I believe $\Sigma_B=\frac{1}{K} \Sigma_{k_i} \Sigma_{k_j} (h_{k_i} - h_{k_j})^2$ would be more correct for the between-class covariance than $h_G$.

[A3] Thanks for your suggestion. $\Sigma_B$ in Eq. (3) measures the sum of distances between all class means and the global mean, while the $\Sigma_B$ you mentioned measures the sum of distances between pairwise class means. These two forms may be influenced by class imbalance, but they are equivalent when the classes are balanced. We simplify the notation $k_i$ and $k_j$ as $i$ and $j$. The proof is provided below.

When the number of each classes is balanced, we have $h_G = \frac{1}{K}\Sigma_{k_j} h_{k_j} = \frac{1}{K}\Sigma_{k_i} h_{k_i}$, The third term above can be transformed into:

Therefore,

In our task, fine-tuning accuracy is measured on the test set. To ensure generalization, the training set is typically class-balanced.

We provide the comparisons of these two methods on the heterogeneous model zoo with a single source (Table 1 in our paper) below. Method "FaCe w/ global cov'' is our $\Sigma_B$, and FaCe w/ btw-distance is your proposed one, the results show that there is minimal difference between the two.