BeST - A Novel Source Selection Metric for Transfer Learning

We thank you for your valuable feedback and apologize for the delay in our response as we were trying to add new simulation results to respond to the comments by all the reviewers.

I think the very weakness lies in the ... lack intuition behind this idea.

Response: We apologize if certain parts are too technical and will try to explain in simple words here.

1. Equation 1 deals with the quantization of a softmax vector to a $q$-level quantized vector. Figure 2 explains the process of converting a 3x1 softmax vector to a 9x1 quantized vector which involves mapping the 3x1 vector to a 2-D grid and then unfolding the grid in row-by-row fashion to get a column vector. Equation 1 is nothing but a fancy way to denote the index i’ in the final column vector that will be 1 while the others are 0.
2. In its simplest form, quantization of a vector [0.9, 0.1] with $q$=4 is simply dividing the number line [0,1] into 4 parts and deciding which part 0.1 belongs to (in this case, the vector will read [1,0,0,0]).
3. The entire idea as to why the quantization approach was introduced in the first place is explained in the next response (while answering the question in questions section of your review, adding as a separate response as word limit reached).
4. The key steps in the proposed method are - quantization, finding a policy for each quantization, finding train and validation accuracy in the analytical form, finding the optimal quantization, and calculating the corresponding metric. Having explained quantization earlier, we can focus on the policy. It is clear that there are multiple possible mappings $\pi^q$ from the quantized source output to the target labels. Each mapping has a train and validation accuracy. Remember these train and validation accuracies here are not an output of traditional training but the performance of the analytical mapping. We need to find a policy $\pi^q_*$ that maximizes the training accuracy (Eq 4). The variation of validation accuracy of this policy $\pi^q_*$ with quantization levels is given in Figure 4. Now, the objective is to find the quantization $q^*$ that gives the highest validation accuracy. The highest validation accuracy is our metric M (Eq 5).

I think in Algorithm 1, it may not be easy to reproduce from the current description.

Response: We apologize if the description in Algorithm 1 is unclear.

1. Refer to Figure 4, where we can see the variation of $A^{val}$ with $q$. Algorithm 1 is simply a ternary search (variation of binary search) on the variable $A^{val}$ using a right and left pointer named $m_1$ and $m_2$.
2. In line 1, we first make sure that the dataset is balanced i.e. equal samples for each class, then the left and right search boundary is defined in line 2.
3. Lines 3-16 are just running basic ternary search to find the mode of $A^{val}$ vs $q$ and the steps inside the while loop is simply the steps to calculate $A^{val}$ for $q=m_1$ and $q=m_2$.
4. Once stopping condition is reached, lines 17-21 are simply steps to calculate $A^{val}$ at the final right and left pointers $L$ and $M$. The metric is the average of $A^{val}$ at these two pointers.
5. Note - Steps 5 and 17, where a quantized version of dataset is created, it simply means that we did a forward pass of the target data through the pre-trained source model and performed the quantization (with level $q$) of the softmax vector to get a one-hot vector.

I suggest the authors clarify some simple concept such as task-similarity, selection strategy, etc. in implementation, without particular wrap up in mathematical aspects.

Response:

1. In simple words, task similarity depends on the capability of the source model to distinguish target data by providing different output representations for different labels. If the source output is similar for target data from different classes, the tasks are not very similar (when we talk about tasks, we mean models as task is an abstract concept). It also depends on the training parameters, transfer architecture, and target dataset. In our paper, we define task similarity or task transferability by the test accuracy achieved when the target model is trained using a frozen pre-trained source model using target data (Figure 1).
2. As explained while replying to weakness 1, for every quantization level $q$, there are multiple mapping policies $\pi^q$ possible that map the quantized source output to target labels. For each quantization level, we want to select a policy $\pi^q_*$ that maximizes the training accuracy and then calculate its validation performance. Finally, we want to search for the optimal quantization level $q^*$ that maximizes the validation accuracies across different such policies. The maximum validation accuracy calculated through this process is the proposed metric M (Eq 5). In implementation (Algo 1), we calculate the metric using a ternary search to find this optimal validation accuracy $A^{val}(\pi^{q^*}_*)$.

We thank you for your valuable feedback and apologize for the delay in our response as it took a while to add new simulation results to respond to the comments by all the reviewers. We have now added 3 sections in the appendix of the paper (F.G.H) (details announced as a separate response).

Based on my understanding, the quantization level ... study on the quantization level.

Response: We believe that there might be some misunderstanding regarding the quantization talked in the paper. The quantization of the softmax values of the pre-trained source model is only used to get our transferability metric (Equation 5) and has no significance in determining the target task performance. The target task performance is defined as the test accuracy when the target model (illustrated in Figure 1), is trained keeping the pretrained part frozen.

As mentioned by the authors, the proposed method relies on early stopping during source pre-training. ... their figures to reflect this variability.

Response: Thank you for the comment. We have added Appendix H - 'Variation with different random seeds for pre-trained source' with the simulation results using 5 different random seeds for source model initialization before training. Specifically, in Fig H.1, we show the variation of $A^{tr}$ and $A^{val}$ (train and validation accuracy) following a similar structure for different random seeds for MNIST-MNIST and CIFAR10-CIFAR10 transfer setups. In Fig H.2, we present the fraction of the correct rank for the CIFAR10-CIFAR10 transfer setup with mean and standard deviation across random seeds. Owing to the limited time frame of the rebuttal, we used these two transfer setups.

The experiments are conducted ... model selection strategy.

Response: Our problem statement for transferability measure primarily focused on classification tasks. Hence, our approach cannot be extended directly to time-series and regression tasks. However, we acknowledge that it is a good point and we will try to work to incorporate it in our next work.

While the manuscript focuses on selecting the best source model, ... handle negative transfer.

Response:

1. A source model in the given set of sources can have varying degrees of transferability for a given target, going from low to high transfer accuracy. The proposed metric is a quick measure of this transferability. When the metric is run for a source that is unrelated to the target, the metric gives a value that ranks it amongst the worst performing in the set. Here, the unrelated sources can be considered as contributing to negative transfer (note that negative transfer is a spectrum as there is no well-defined threshold to be called negative transfer).
2. To show that our proposed metric handles negative transfer cases well, refer to Figure 5, where the metric values (blue) and the transfer learning accuracy (red) are represented for different source models for 3-class source to binary target tasks for MNIST-MNIST transfer setup. We can see that source tasks with the worst transfer performance (~60% accuracy) like source to classify digits (0,3,8), (0,1,5), (0,5,8), etc., have their corresponding metric values very low indicating bad transfer performance.

Certain theoretical assumptions, such as how the ... with different underlying tasks.

Response: This is an initial work for this problem statement where we wish to demonstrate a novel quantization-based approach. Studying the theoretical aspects of the relationship of the early stopping point with the generalization of the target model is our aim for future work. However, intuitively, there is a relation between the accuracy of the trained target model and the metric value (calculated without training) since in both cases, we try to maximize the training performance (accuracy/loss) while monitoring the validation behavior. We stop when the validation performance degrades, which is common in both setups, the difference being we monitor validation performance over time in traditional training but over quantization levels $q$ in metric calculation.

Early stopping is a well-studied method for ... literature review on early stopping.

Response: We wish to clarify that we do not aim to propose a new early-stopping method. We use a very commonly used early stopping method i.e. stop when validation performance (accuracy in our case) starts decreasing. Our approach uses this concept of early stopping to select an optimal quantization level $q^*$ (unlike the epoch for when to stop in traditional early stopping). The novelty is to use this well-known concept in a different problem i.e. problem to choose an optimal quantization level. Therefore, a direct comparison between traditional early stopping baselines and our method is not applicable.

How sensitive is the method to changes ... might not be feasible?

Response: Thank you for your question.

1. Interpreting sensitivity as the change in $A^{val}$ as quantization level $q$ changes, we can say that it depends on several factors like the quality of the pre-trained source (related or unrelated to target) and the target data. For source models with low transfer performance ($\sim$50% accuracy), the curve for $A^{val}$ vs $q$ (similar to the one illustrated in Fig 4) starts around 50% and remains around the same as $q$ changes with small zig-zag-like variations. However, for good source models (transfer accuracy of $\sim$80% or more), we can see there is a large change in $A^{val}$ with a small change in $q$ at first (as in Fig 4) and after the optimal $q^*$, the decrease in $A^{val}$ is rather slow. Hence, the sensitivity varies.
2. By definition of our method (Eq 5), it is always possible to find an optimal $q^*$ as we can always run both a brute force search and a ternary search (Algorithm 1) on the fixed search space for $q$.

Can the proposed method be applied to tasks beyond classification, such as regression?

Response: This paper addresses the problem setup for transfer learning in classification tasks and was not developed with regression tasks in mind. Therefore, applying our method directly to regression tasks would be challenging.

We hope that our detailed comments and the additional data (new appendix sections of the paper) have provided clarity on the questions and concerns raised regarding our paper. We humbly request the reviewer to kindly reconsider their evaluation in light of the updated information, should they find it appropriate.

None author response found. I will keep my score.

We thank you for your valuable feedback and apologize for the delay in our response as we were trying to add new simulation results to respond to the comments by all the reviewers. We have now added 3 sections in the appendix of the paper (F.G.H) (details announced as a separate response).

The experiments are conducted using only DNNs ... datasets like ImageNet.

Response: Thank you for your feedback.

1. The goal of this work was to showcase a proof of concept for the proposed novel quantization-based approach. To achieve this, we utilized smaller datasets, allowing us to produce comprehensive results efficiently within a shorter timeframe.
2. We have now added additional simulation results for the Imagenette dataset (a subset of Imagenet) with 64x64 images in Appendix G where both the source and target tasks are binary classification tasks for images in the Imagenette dataset. We have included plots to visualize the source rankings given by our metric and compared them to the true rankings in Section G.1. In G.2, We have results for the fraction of accurate rank predictions for two different target tasks. Finally, in Section G.3, we present the time improvement statistics for using our metric versus training the model to find transfer performance.

The paper fails to include a comprehensive comparative analysis with contemporary methods such as NCE, LEEP, and LogMe. ... selection in transfer learning.

Response:

1. Methods like NCE (Tran et al. (2019)), OTCE, and LEEP (Nguyen et al. (2020)) use both the source and target labels to estimate transfer performance. Our approach is different in that it is a very lightweight approach that does not need the source data used to train the source model. This makes our work practically appealing compared to the state-of-the-art. It also means that it is not possible to have a fair comparison of our approach with methods like NCE, OTCE, and LEEP.
2. Our setup differs from LogME in that we only have access to the softmax outputs of the pre-trained source model, as we operate under a black-box source setup (justification provided in response to weakness 3). In contrast, LogME explicitly states that it does not utilize the classification head (softmax outputs). Therefore, a direct comparison between our approach and LogME is not feasible.

The manuscript assumes the use of a fixed source ... strengthen the manuscript's relevance and applicability.

Response: Thank you for this question. We envision that with the development of closed-source pre-trained models like ChatGPT, Claude, etc., there will be increase in the availability of private models which can be only accessed through an API (without knowledge of the parameters) and we can only access the probability distribution over the output labels (i.e. softmax output). In such a case, to select the most transferable source, it is practical requirement to evaluate the transfer performance of the source for a given target task. Our proposed transferability measure is aimed to target such scenarios.

Measuring the relatedness between the source and target tasks to ... benefits over previous strategies.

Response:

1. In their work in [1], as detailed in Section 3 of their paper, the authors focus on a transferability measure that requires evaluating transfer performance by training a target model. The task taxonomy they propose, represented as a directed hypergraph, is constructed based on these transfer performances. In contrast, our work aims to eliminate the need for training the target model to assess transferability. Hence, the objective of our work is different and it's difficult to have a fair comparison between both.
2. As mentioned in the response for weakness 2, existing methods for transferability estimation like OTCE [2], NCE assume that the source data that was used to train the source model is available. Our approach is aimed to study a different setting which does not need the source data, making it lightweight as it only needs the pre-trained source model.
3. In their work in [3], the authors assume that the probability distribution of the reference task (source) $P_R(x,y)$ is known. However, this contradicts our setting as we do not have access to the source data and hence would not have this distribution available. Our work only uses the distribution of the softmax output of the source given target label.

Thank you for your response. I appreciate the clarification provided, but my concerns remain unresolved. Specifically, the unconventional black-box source model setting and a lack of comprehensive comparison with existing methods.

"These papers do not use the word 'black box' while still using a similar approach to ours."

This statement reflects another misunderstanding. When metrics like LEEP or NCE are computed, the source pre-trained model is indeed fixed (like a black box). However, my concern lies with the fine-tuning process. In the referenced papers, the entire model is accessible, allowing for full fine-tuning (tune all parameters) with target data. Conversely, in this manuscript, the black-box setting only adds additional linear layers while freezing the pre-trained model. This limitation can significantly compromise performance.

I strongly recommend that the authors revisit the NCE, LEEP, and LogMe papers. Proposing a good metric for linear fine-tuning—where only one or a few linear layers are updated—is not practically useful, as linear fine-tuning is computationally lightweight and can be done very quickly. In fact, the time required to compute a metric may sometimes exceed the time required for linear fine-tuning itself [3]. Thus, there is little practical value in computing a metric instead of simply assessing the linear fine-tuning performance.

A useful metric should correlate strongly with the performance of full fine-tuning (where all model parameters are updated), as full fine-tuning and hyperparameter tuning are computationally expensive. This is also why evaluating the computational time on GPUs is necessary to establish the method’s efficiency.

"The evaluation of whether training from scratch is better does not align with the core idea of the paper."

This critique is closely tied to the unrealistic nature of the black-box setting. The rationale behind training a classifier on a model that cannot be fully accessed is unclear. Unlike black-box generative models, there are numerous publicly available large foundation models, such as CLIP, which already offer strong performance.

Additionally, I searched for Clarifai and found:

"Clarifai provides an end-to-end, full-stack enterprise AI platform to build AI faster, leveraging today's modern AI technologies like cutting-edge Large Language Models (LLMs), Generative AI, Retrieval Augmented Generation (RAG), data labeling, inference, and much more."

This suggests that Clarifai is not a provider of black-box classification models but rather an AI development platform. The manuscript should include a clear and practical use case for the proposed black-box setting to justify its relevance.

"Nonetheless, our clarification on applicability for [1] [2] [3] (as in your original comment) remains the same."

I disagree. The assumption and experimental setup for computing the metrics are the same across the referenced methods, so they are indeed comparable. A thorough comparison with existing methods under same experimental conditions (e.g., datasets, model zoo) would significantly strengthen the paper.

In conclusion, the paper still requires substantial improvements to address these concerns and provide clarity on the proposed setting and method. Additionally, the sluggish rebuttal process has left insufficient time for further discussion. Therefore, I will maintain my initial score.