Understanding the Transferability of Representations via Task-Relatedness

Thank you for your thoughtful review. We are delighted that you found our analysis clear and method practically valuable. We respond to your concerns below:

1. Limited novelty: (A) the idea ... is not new. (B) the proposed method ... justification.

A: While we agree with the reviewer that distribution distance is effective at gauging transferability, prior works such as OTDD and OTCE offer limited theoretical justification to explain a model’s transferability. As mentioned by the reviewer in their paper’s summary, our analysis shows that there are two additional terms (namely, reweighted reference loss and conditional entropy of label sets) along with distribution distance to provably explain transferability. Moreover, unlike OTDD and OTCE which rely on the distance between the distributions of the reference and the target tasks, in the input and representation space, respectively, our analysis shows that the distance between the transformed distribution of the reference task in the representation space (i.e. P_{R’’’} obtained after using the transformation A) and the distribution of the target task is what explains transferability provably (Theorem 3).

B: We agree with the reviewer that task-relatedness can estimate transferability without requiring target labels easily and we emphasize that this ability is unique to our approach. This is because, to the best of our knowledge, no other transferability estimation method can produce pseudo-labels of the target data in the label space of the target task (Methods such as LEEP [34] rely on pseudo-labels of the target data in the label space of the source task). Using these labels for the target task, we can compute task-relatedness as per Def. 2 and still estimate the transferability of the models (Tabel 2). This makes task-relatedness the only method that can gauge the transferability of models without access to target labels.

Thus, compared to prior works, our paper advances both our theoretical and practical understanding of transfer learning in a meaningful way.

2. Limited evaluation: (A) OTCE is ... not discussed. (B) no comparison ... pseudo-labeling schemes (C) no ablation study ... task-relatedness.

A: Please see our joint response for comparison with OTCE.

We thank the reviewer for pointing us to [a] and we briefly compare our approach to them here. We will include this discussion in the main paper.

[a] proposes a model-agnostic approach that relies on optimal transport to compute the distance between the tasks similar to OTDD [3]. However, our work specifically focuses on understanding the transferability of different models both theoretically and practically and hence proposes a model-dependent metric for it.

B: As mentioned above, no previous score-based transferability estimation methods can work without access to labels of the target data. Methods such as LEEP [34], use pseudo-labels of the target data from the source classifier but still require labels of the target data to compute their transferability score. Thus, to the best of our knowledge, task-relatedness is the only approach that can work when no target labels are available. Due to this, we show its effectiveness by comparing the correlation of task-relatedness with and without target labels to actual transferability in Table 2. Our results show that even without target labels, task-relatedness achieves a good correlation with transferability when reference and target tasks are related.

C: In Appendix C.1, we study the effect of different transformations on task-relatedness. We present three settings, the first where no transformations are learned, the second where feature transformation is learned, and the third where all the transformations are learned. Our results show that task-relatedness achieves the smallest gap to transferability when all transformations are learned, closely followed by learning only the feature transformation, and significantly better than not learning any transformation. These three settings capture the effect of the three different transformations proposed in our transformation model. We also break down task-relatedness into its three corresponding terms in Figs. 3 or 4b to highlight the contribution of each term.

1. Class-prior ... assumption?

Please see our joint response.

2. In Figure 3, ... fixed here.)

The full results of Fig 3 presented in Fig 9 (in the Appendix) already include models trained with different model backbones such as ResNet-18, ResNet-50, ResNet-101, and ResNet-152 trained with a variety of training algorithms. Along with these results, Fig 4(a) shows the effect of training the same model backbone on different reference datasets and evaluates transferability to a variety of target tasks. Fig 4(b), evaluates transferability to various target tasks using the original pre-trained CLIP encoder and CLIP encoder fine-tuned end-to-end on different reference tasks. Thus, our experiments already evaluate the effect of different model backbones and reference datasets on transferability and task-relatedness.

3. Application ... auto-regressive prediction?

Please see our joint response.

4. What's the ... holds?

Similar to any other upper bounds that analyze the problem of distribution shift (e.g., [6]), a large upper bound implies that theory might not be able to explain the results in practice. This is true for task-relatedness as well since a large upper bound does not necessarily guarantee poor transferability. Using various datasets/tasks (in Fig 3 and 9) we show that task-relatedness estimates transferability closely. Moreover, under the conditions mentioned in Lines 221-225, our bound is indeed tight.

New References

[a]: Wasserstein Task Embedding for Measuring Task Similarities

Thank you for your thoughtful review. We respond to your concerns below:

Compare with [1,2,3,4,5]

We thank the reviewers for pointing us to additional related works. We briefly compare our approach to them here and will include the discussion in the main paper.

[1,3,5]: study the problem of few-shot learning (FSL) where a model is trained on data from related training tasks and is adapted to an unseen task using only a few samples. Different from these works, we focus on the transfer learning (TL) setting where a pre-trained encoder trained on some pre-training dataset is adapted with enough samples/gradient steps to a downstream task. This downstream target task may or may not have any relation to the pre-training task unlike [1,3,5].

Concretely, [1] proposed a model-agnostic metric called Task Attribute Distance to gauge the success of learning in the FSL setting. Our work, on the other hand, defines task-relatedness based on the similarity of the representations of reference and target tasks in the representations of the pre-trained model (and is model dependent) rather than relying on the attribute information, which may not be available in the TL setting.

[2] analyzes the sample complexity for learning a model shared across tasks and adapting it to a new target task and shows task diversity to be a crucial component for the success of FSL. Our work on the other hand does not assume access to shared tasks or restrict the number of samples required for fine-tuning on the target task. Moreover, their notion of task diversity requires access to a set of training tasks that may not be available in the TL setting, making our notion of task-relatedness more practical for TL.

[3] proposes a notion of task-relatedness for the FSL setting, allowing to utilize all the data from available training tasks to help learn a model on a new task with a few gradient steps. This notion is model-agnostic and defined over the sample space ($X \times Y$) unlike our measure which is defined in the representation space of the model whose transferability needs to be evaluated.

Thus while task-relatedness is at the core of both TL and FSL, the works [1,3,5] proposed notions relevant to the FSL setting whereas our work proposed a notion relevant to the TL setting.

Please see the joint response empirical comparison against [2,4].

Although this work ... baselines on transferability.

Ablation studies: In Appendix C.1, we show the effect of using different transformations on the value of task-relatedness computed by solving Eq. 3. We present the results in three settings, the first where none of the transformations are learned, the second where only the feature transformation (parameterized by A) is learned, and the third where all the transformations are learned. The experimental results show that task-relatedness achieves the smallest gap to transferability when all transformations are learned, closely followed by learning only the feature transformation, and significantly better than not learning any transformation. These three settings capture the effect of the three different transformations proposed in our transformation model.

Trade-off experiments: In Appendix C.1.2, we evaluate the number of epochs required by Alg 1 to minimize the proposed upper bound (compute task-relatedness) for four target tasks using the ResNet-18 model and Imagenet as the reference task. Our results in Fig 8 show that after approximately 600 epochs (approximately 2 minutes of wall clock time on our hardware) Alg. 1 learns the transformations that lead to a good estimate of transferability.

For the problem of end-to-end transferability estimation (Sec 4.3) for different models, we find that the computation of task-relatedness is significantly faster than the computation of actual end-to-end fine-tuning. Specifically, the computation of task-relatedness requires roughly only 3-4 minutes for computation, compared to end-to-end fine-tuning which can require computation of several hours. E.g., You et al., 2021 [54] show that end-to-end fine-tuning of a ResNet-50 model on the Aircraft dataset, with hyperparameter search, requires about a day’s worth of computation to achieve the best accuracy of 86.6% (see Sec 5.5 and Table 5 of You et al., 2021). In comparison, task-relatedness can be estimated very efficiently. Compared to other SbTE approaches such as SFDA, our approach increases the computational time by ~30 seconds but gets consistently better results as shown in Table 1 and Fig 5 of the paper and the pdf in the joint response.

Comparisons with baselines on transferability: In Fig 3 and 4(b), we compare the estimate of transferability computed via Alg. 1 to the ground truth value of transferability (denoted by the blue bar labeled target loss). Our results show that task-relatedness incurs a small error compared to actual transferability.

In Table 1, we present the result of comparing the correlation of task-relatedness and accuracy after end-to-end fine-tuning and compare it to the correlation of five popular score-based transferability estimation methods. Along with these we also compare the effect of the number of target labels available on the correlation of each of these methods with the accuracy after end-to-end fine-tuning.

Thus, we believe that we have already provided a detailed empirical evaluation of our methodology and compared our approach with popular transferability methods. However, if the reviewer would like to see any specific experiments to be added, we would be happy to provide them as well.

The formatting ... definitions.

We thank the reviewer for this excellent suggestion. We will create a table to clarify all the symbols and their meaning. We will expand and adjust the tables and equations to be more readable in the camera-ready version.

We thank the reviewer for their response to the rebuttal.

While we agree that transfer learning is a broad research topic, our paper specifically works in the inductive transfer learning setting where the focus is to leverage an inductive bias (a pre-trained model) to improve performance on a target task. Due to this, our analysis specifically proposes a model-dependent way of measuring the relatedness between a reference and a target task. The works such as [1,3,5] provided by the reviewer deal with the problem of task transfer learning. Their goal is to identify the relationship between tasks, regardless of the model, to explain the transfer performance.

E.g., the task attribute distance proposed by [1] is independent of the model being used and only depends on the tasks. (This terminology of inductive and task transfer learning is based on a previous work LogMe[54] which has the same setting as ours.). Thus, the setting of our work differs significantly from the setting in [1,3,5].

We thank the reviewer for bringing this up. We will add a discussion to the paper and clarify the type of transfer learning being studied in the paper along with the mention of the relevant papers from the area of task transfer learning.

We would be happy to address any further concerns the reviewer may have.

Thank you for your thoughtful review. We are glad that you found our work intuitive. We respond to your concerns below:

The datasets for evaluation and empirical studies seem to be small in scale, for example, MNIST and CIFAR which is small in the image resolution, or Aircraft and DTD which is small in the number of images.

The datasets used in our work are the ones used by prior works. This was done to make the comparison with previously proposed methods for transferability estimation easier. Moreover, most model backbones used in our work require data to have dimensions 224x224, so we resized images from all datasets including CIFAR to match those dimensions. This shows that our method handles data with a larger resolution.

My only question is that for algorithm 1, does this has guaranteed convergence? and practically how long does it take to reach convergence?

Similar to other problems in deep learning this is a non-convex minimization problem where SGD converges to a local minima. Fig 8, in Appendix C.1.2 shows how Alg. 1 minimizes the proposed upper bound by learning different transformations of the reference task (ImageNet) into four target tasks (CIFAR-10, CIFAR-100, Pets, and Aircraft) with the ResNet-18 model. After approximately 600 epochs (approximately 2 minutes of wall clock time on our hardware) the Alg. 1 converges to local minima.

To demonstrate that Alg. 1 converges to a reasonable solution, we consider transfer learning using a 20-class subset of Imagenet as the reference task and CIFAR-10 as the target task. Results in Fig. 6 (bottom left) and Fig. 7 (in the Appendix) provide clear evidence of Alg. 1 finding the transformations that minimize the upper bound.

Specifically, without learning any transformations the upper bound is large (~ 3.22 in Fig. 6) and the reference and target classes do not match (leftmost plot in Fig. 7). However, learning all the transformations reduces the upper bound (3.06 in Fig. 6), suppresses the prior of 10 source classes to match the 10 classes in CIFAR-10 and reduces the Wasserstein distance between the transformed source and target classes (rightmost plot in Fig. 7) illustrating the working and convergence of the optimization problem to a good solution.

Compare with Discriminability-Transferability Trade-Off: An Information-Theoretic Perspective

We thank the reviewers for pointing us to this work. We briefly compare our approach to it here and will include the discussion in the main paper.

The work demonstrated a tradeoff between a model's discriminability and transferability properties as training progresses and proposed a learning framework to alleviate this tradeoff. Our work on the other hand focuses on analyzing the transferability in terms of relatedness of the representations of the reference and target tasks after training is complete.

Thank you for your thoughtful review. We respond to your concerns below:

Utility: A) I was ... methods. B) Specifically, ... target task.

A: While Alg. 1 requires labels of the target task, we discuss in Lines 329-347 how to use Alg. 1 for the case when target labels are unavailable. The main idea is to use the transformation model to obtain the pseudo-labels (Line 337) of the target data in the label space of the target task. (Note that this is different from the pseudo-labels of the target data in the label space of the source task as used by LEEP [34]). Using these pseudo-labels of the target data as a proxy for $Y_T$, we can now use Alg. 1. Since no other SbTE method (to the best of our knowledge) can work without target labels, we stated this in the introduction of the paper. We will update the input of Alg. 1 appropriately to clarify this.

B: We emphasize that the reference task does not need to be similar to the target task to use our methodology for estimating transferability. Any benchmark/publicly available dataset can serve as the reference dataset and its use is not a bottleneck for our analysis/approach. Instead, it makes it possible to study transferability provably, enables us to estimate transferability without target labels, and provides insights into improving the transferability of a model to downstream tasks.

None of ..

While the best way to evaluate transferability is to fine-tune a model end-to-end on the target task, it is computationally prohibitive. E.g., You et al., 2021 [54] show that end-to-end fine-tuning of a ResNet-50 model on the Aircraft dataset, with hyperparameter search, requires about a day’s worth of computation (Sec 5.5 and Table 5 of You et al., 2021). This is the motivation for most SbTE approaches that efficiently estimate transferability without end-to-end fine-tuning.

Previous works [5, 34, 25, 54, 49] reported the Pearson correlation coefficient between the accuracy of models after end-to-end fine-tuning and the proposed scores. A high correlation of an SbTE score indicates that comparing the scores of various models and choosing the one with the highest score is most likely the best-performing model on a given target task. Thus, we report correlation coefficients. (Since task-relatedness is based on loss, a higher negative correlation is desirable for us.)

[a]. From line 238 - 241..

We ablate the use of different transformations and measure the effect of learning each of them in Appendix C.1. Our experiments show that minimizing the conditional entropy by learning transformation B prefers a sparse mapping between the classes of the reference and the target tasks. As a result, one class from the target is mapped uniquely to one class from the reference task, and the priors of extra reference classes are reduced to zero (This is illustrated in Fig. 7 of Appendix C.1.1). Based on this, we use a reference task with the same number of classes as the target task and fix the matrix B to be a random permutation of the identity matrix.

Thus, while each term in the bound has its impact, fixing the matrix B and choosing a reference task with the same number of classes as the target, we simplified the optimization problem in Eq. 1 and made it efficient for practical usage.

[b]. It feels..

The main purpose of the experiments in Sec. 4.2 is to demonstrate the validity of our analysis and the fact when the target task is a transformation of the reference task, a model’s transferability to the target task can be provably explained based on that of the reference task. For e.g., for an encoder trained on MNIST, ground truth transferability to USPS is better than that to FMNIST (Fig. 4(a) right). This is also what is suggested by our task-relatedness analysis/metric using MNIST as the reference task (Fig. 4(a) left). Thus, as mentioned in Fig 1, given a pre-trained encoder and a reference task, we can provably explain the encoder’s transferability to those target tasks that are transformations of the reference task.

For the practical purposes of estimating transferability on the other hand, any reference task can be used (i.e., any publicly available or benchmark data) to compute task-relatedness. As shown in Sec 4.3, the use of a reference task to estimate transferability leads to a more stable estimate of transferability regardless of the number of target samples used as well as enables estimating transferability without access to target labels. Thus, while being theoretically sound, task-relatedness provides practical advantages over previous transferability estimation methods.

[c] The authors..

A small gap in transferability implies that the difference between the left-hand side of the bound in Theorem 3 and the right-hand side computed by learning the transformations via Alg. 1 is small. While we agree that losses at random initialization for reference and target loss might show a small gap, it will not be very useful. Thus, we measure the left-hand side of the bound after training on the target task and then compare task-relatedness with this value. The target loss presented as the blue bars in Figs 3 and 9 show this value.

The $K_r > K_t$ ... when $K_r < K_t$ ?

Please see our joint response.

classification based ... generative tasks

Please see our joint response.

The analysis..

This seems to be a misunderstanding since our analysis is for expected loss. The classifiers $h_R$ and $h_T$ are learned on the training data and we report transferability and task-relatedness values on the test data from the two tasks similar to any ML pipeline.

When there is overfitting on the reference task, we can expect the reweighted reference task loss (computed on test data) to be higher, which may lead to a larger gap between the actual and predicted transferability. We emphasize that this is not unique to our method as any analysis based on a source/reference task (e.g., [6,7]) will behave similarly.

We thank the reviewer for their response to the rebuttal.

Are ... of B.

The reviewer’s intuition is correct that the pseudo labels are computed using the $B$ matrix provided at the initialization of Alg. 1 for the first time. But as Alg. 1 progresses the matrix $B$ will also be updated. As a result, the pseudo labels of the target data are also updated. This can be thought of as an additional step in Alg 1 as follows:

Step 2a: If target labels ($y_T^j$) are not available then compute $y_T^j = arg max_{y \in Y_T} Bh_R(A^{-1}x_T)$ for all $j = 1, …, n_T$.

For computing the results in Table 2, the matrix B is fixed to a permutation matrix (similar to our other experiments). Thus, there is no variability due to the matrix B.

I would ... mistaken.

The reviewer's intuition is correct. The choice of reference task affects the upper bound as we demonstrated, in detail, in Sec. 4.2. Specifically, when the reference and target tasks are more related (in terms of our task-relatedness metric) the upper bound is smaller.

However, finding the most related reference task for every target task may not be possible in practice. Since our analysis does not impose any restriction on the choice of the reference task to use for computing task relatedness, a practitioner can estimate the upper bound using a few benchmark/publicly available datasets and select the reference task as the one that minimizes the upper bound, similar to the approach suggested by the reviewer of treating the reference task as a variable.

I would like ... problem.

This seems to be a misunderstanding. SbTE methods as per [54] produce a score for each model (ideally without end-to-end fine-tuning on the target) that correlates well with end-to-end accuracy. This allows selecting the top-performing model by simply evaluating these scores. Thus, all prior works report correlation coefficients similar to our results in Table 1.

Moreover, scores produced by SbTE methods do not numerically approximate the accuracy after end-to-end fine-tuning. For example, the results in Fig. 4 of [54] show that the value of SbTE metrics (y-axis of the plot) such as LEEP [34], NCE [50], and LogMe [54] lies in the range [-0.8,-0.3], [-0.45, -0.25], and [0.935, 0.95] respectively for the Aircraft dataset whereas the end-to-end fine-tuning accuracy (x-axis of the plot) lies in the range ~[72.5, 87.5] for the various models.

It would be great if the reviewer could clarify their question if we have misunderstood them.

It is very ... suffice.

The experiment in Appendix C.1 is a small-scale experiment as it is in the same setting as in Sec 4.1 with the ResNet-18 model. It is intended to be an ablation study to understand how different transformations affect the upper bound.

The label mismatch term of Theorem 3 is minimized with a sparse B matrix corresponding to a one-to-one mapping between the classes of the reference and target tasks. However, due to the combinatorial nature of the problem finding the exact permutation is infeasible. Regardless of that, the linear transformation $A$ can align the distributions of the representation of the target to any permutation of the reference classes quite accurately. This is primarily due to the high dimensionality of the representation space where the linear transformation is applied. As a result, the value of the bound differs only by little for the different permutations of the reference task’s classes. This is indeed a very interesting outcome that makes our bound usable without requiring the exact semantic matching between the classes of the reference and the target classes.

Below, we present an empirical evaluation to show that the difference in the bound computed with true vs random permutation is small. We use MNIST as the reference task and USPS as the target task (and vice-versa). We compare our results in a setting where only $A$ is learned and $B$ is set to an identity matrix and when $B$ is set to a random permutation matrix. Note that the identity matrix corresponds to the correct mapping between the classes of MNIST and USPS tasks (both contain digits from 0 to 9).

We find that the upper bound obtained when $B$ is fixed to identity is only marginally better than the case when $B$ is a random permutation. Specifically, the difference between the bound when $B$ is fixed to a random permutation and when $B$ is an identity matrix is 0.10 for the MNIST→USPS task and 0.17 for the USPS→MNIST task. The primary reason for the decrease in the upper bound comes from the reduced distribution mismatch term.

While the upper bound improves slightly when the ideal matching between the labels is known, such a mapping may not be known when the labels of the tasks are not related, e.g., for FMNIST and MNIST. Thus, fixing $B$ to a random permutation matrix yields a reliable estimate of transferability in most cases.

We would be happy to address any further concerns the reviewer may have.

Thank you for your interest in our work; we appreciate your questions. We have answered your questions below. Please feel free to reach out if you have any further questions.

The proposed ... this?

Our work's primary contribution is understanding on which tasks transfer learning can lead to high-performing models. To answer this question, we consider a reference task that achieves high performance after fine-tuning. Through our upper bound in Theorem 3, we show that performance on any target task can be provably related back to this reference task. Crucially, the reference task may or may not be the source task used to train the pre-trained model, and any publicly available dataset can be considered the reference task. Lastly, various previous transferability estimation methods such as NCE[56] and OTCE [55] have also utilized the source task for computing their metric for transferability.

In our empirical section, we show that task relatedness is also an effective metric for the pre-trained model selection problem. We also emphasize that, unlike our work, an analytical understanding of transferability in the cross-domain cross-task setting is not the focus of other score-based transferability estimation methods.

How ... unavailable?

As mentioned above, our analysis does not restrict the choice of the reference task to use for computing task relatedness. Any benchmark or publicly available dataset can serve as the reference task.

Even if ... your opinion?

In Figures 3 and 9, we show experiments with various self-supervised models such as MOCO, SWAV, and MAE. We used Imagenet as the reference task for these experiments. Since the reference task can be any task, all terms in the bound are still meaningful.

From Table 1 and Figure 5, ... as a practitioner?

We see in Figure 5 that LEEP performs worse in scenarios where there is less target data available. Comparatively, our method produces consistent correlation at different percentages of target data.

While we agree that LEEP indeed offers the practical advantages you mentioned, it is sensitive to the amount of target data available for transferability estimation, and it only provides limited insights into analytically explaining transferability, unlike our approach.

A) The paper mentions ... of their method? B) Further, how this step can... class labels?

A) While we agree that practically various alternatives such as off-the-shelf clustering methods could be used to obtain pseudo-labels, our task transfer approach does not require using these to handle unsupervised transferability estimation, unlike other score-based methods.

B) For pre-trained SSL models, the predictions from $h_R$, which is the classifier trained on the reference task, can be used to obtain the pseudo-labels using $\arg \max_{y \in Y_T} Bh_R(A^{−1}(z_T))$ as mentioned in Sec. 4.3.

The claim, ... What is your insight?

While end-to-end fine-tuning depends on various factors, an in-depth analysis from previous works such as Table 4 in LogMe [61] has shown that it is extremely costly to find the best end-to-end fine-tuned model for a single target task along with the best hyperparameter search. This has been the basis of various score-based transferability estimation approaches we cited in our paper.