Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuning

We thank the reviewer for the valuable comments and address the questions below.

Consistency to similarity and diversity to coverage

We formally show consistency is reduced to similarity and diversity to ellipsoid coverage in Gaussian distribution. We consider each latent class/representation as a feature vector. The non-zero entries contain the information/feature. The non-zero entries contain the information. Both consistency and diversity can be depicted through the variations in the entries of target tasks' latent classes, which are encoded by entries in the finetuning tasks' latent classes.

To meet consistency, the latent classes in finetuning classes should not include an excessive amount of unrelated varying entries. This is directly related to the similarity in the context of Gaussian distributed features. In our framework, consistency is quantified by how closely the task-related feature distributions align, which is effectively captured by the similarity between mean vectors.

Similarly, to meet the diversity criterion, we would want latent features in target tasks covered by that in finetuning tasks. Under our theoretical model, diversity equates to the spread of the feature vectors across the task space. The ellipsoidal coverage represents the extent to which the feature space of the finetuning tasks encapsulates the feature space of the target task.

Stronger baselines besides direct adaptation and full finetuning, e.g. meta-learning algorithms

Our experiments were designed to demonstrate the effectiveness of the proposed method against common approaches like direct adaptation and full finetuning. We interpret that the reviewer is pointing to Table 2 regarding our multitask finetuning approach. It's important to recognize that our multitask finetuning strategy falls under the category of meta-learning algorithms, as it involves treating target few-shot tasks as individual tasks and creating analogous few-shot finetuning tasks to refine the model. This process assists the model in learning to tackle these few-shot challenges effectively. In response to the feedback, we aim to incorporate additional meta-learning algorithms into our comparison table to ensure a more equitable evaluation.

We incorporated the Model-Agnostic Meta-Learning (MAML) algorithm, as outlined by Finn et al. [1], as another baseline for our few-shot tasks.

MAML operates in a two-step process: it initially updates parameters based on within-episode loss (the inner loop), then it evaluates and updates loss based on learned parameters (the outer loop). We follow the pipeline in [2] to implement MAML for few-shot tasks.

Here we show the results of the comparison for DINOv2 ViT-S on tieredImageNet. We refer the reviewer to Table 14 in Appendix G.8 for complete results.

As illustrated in the table, MAML can slightly outperform Adaptation and Standard FT for DINOv2 ViT-S on tieredImageNet, but lags behind our multitask finetuning. The reason is in few-shot tasks, support samples (shot image) are very limited, preventing the model from learning a good set of parameters for adapting to new tasks.

[1] Finn et al. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. PMLR 2017.

[2] Triantafillou et al. Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples. ICLR 2020.

We thank the reviewer for the valuable comments and address the questions below.

Gaussian distribution assumption

Please see the global rebuttal.

Relation between $\mathcal{T}_{z,z’}$ and $\mathcal{T}_i (i = 1,\ldots,m)$

We realize that the notation used in Section 3.1 could be made clearer. In Figure 1, we denote each task containing two latent classes, namely ($z$,$z’$). Each task in Figure 1 can be represented as ($T_1$ to $T_{z_1,z_1^\prime}$, $T_2$ to $T_{z_2,z_2^\prime}$). We thank the reviewer for pointing this out. We have updated this in the new revised manuscript.

Confusions in the Experiment Protocol in Section 4

The paragraph Experiment Protocol outlined at the beginning of Section 4 applies to all three major experiments (Sections 4.1, 4.2, and 4.3). We clarify further here: In Section 4.1, we retain the same finetuning setup and vary the sample complexity in tieredImageNet. In Section 4.2, we continue with the same finetuning process and apply the task selection algorithm on Meta-Dataset. In Section 4.3, we retain the same finetuning setup and include the additional standard finetuning where we append the encoder with a linear head to map representations to class logits and finetune the whole model.

All experiments focus on evaluating models using few-shot tasks. For multitask finetuning, the model's parameters are updated using the nearest centroid classifier. This involves encoding all samples, calculating class centroids, and using the cosine similarity between a query sample and these centroids as the class logits. For adapting to a specific target task, we employ the model's encoder along with the same nearest centroid classifier approach.

We will ensure that the protocol is clearly described for each experiment in the revised manuscript to avoid any confusion.

Multitasks for finetuning in Section 4.3

In Section 4.3, the multitasks are the few-shot tasks consisting of $K=15$ classes with $N=1$ support samples and $Q=10$ query samples per class (known as $K$-way $N$-shot). The multitasks selected for finetuning were chosen based on their relevance and consistency with the target tasks, as prescribed by our Consistency-Diversity Task Selection algorithm.

We thank the reviewer for the valuable comments and address the questions below.

Novel insights of theoretical bound

While our derived bounds may coincide with what has been empirically observed, it's important to recognize that such empirical observation, until formalized, remains a hypothesis. By providing a rigorous theoretical foundation, we not only validate these empirical observations but also establish a framework upon which future hypotheses can be tested and understood.

Within the theoretical community, various concepts of diversity have been proposed to establish guarantees [1,2]. Yet, we propose an additional notion of consistency, highlighting the significance of aligning finetuning data with target data. Conversely, practitioners in the empirical community have predominantly concentrated on algorithms whose selection is based on similarity ([3,4] and many more). Our algorithm, which integrates both diversity and consistency, aims to enhance task selection when adapting foundation models to specific tasks.

[1] Tripuraneni et al. On the theory of transfer learning: The importance of task diversity. NeurIPS 2020.

[2] Zhao et al. Blessing of Class Diversity in Pre-training. AAAI 2023.

[3] Liu et al. Learning customized visual models with retrieval-augmented knowledge. CVPR 2023.

[4] Liu et al. What Makes Good In-Context Examples for GPT-3? arXiv 2021.

Gaussian distribution assumption

Please see the global rebuttal.

Evolved $\phi$ in Algorithm 1

We appreciate this insightful observation. Currently, our algorithm does not perform repeatedly during training. Instead, we utilize a pre-trained $\phi$ as a feature extractor to obtain the representation of each individual sample. Online update $\phi$ is interesting and we leave it as a future work.

Difficulty in readability

Thank you for pointing out improvements in the presentation. To aid in the understanding of task selection, we further include a diagram in Figure 2 in Section 3.2 to show relation between tasks. We have a plan following to improve the readability:

• Example illustration: We will hold a simplified example before introducing new notations, including the definition of tasks and loss in Section 2.
• Intuitive explanations: We will introduce each major theoretical result with an intuitive explanation, including our main theorem and theorem for diversity and consistency in Section 3.
• Symbol simplification: We will try to simplify the notation by reducing the number of symbols and by using more familiar notations, including different models $\phi$, task $\mathcal{T}$, loss $\mathcal{L}$ in Section 3.

We hope these improvements will enhance the clarity and make our paper more approachable to broader range of readers.

Performance decreases with the increase of shots in Figure 2a

We illustrate this phenomenon below as we mentioned in paragraph Results. in Section 4.1. In Figure 2a we fix the target task as a $1$-shot setting but vary the number of shots from $1$ to $4$ in finetuning. We recognize that the results may initially appear counterintuitive as they oppose the common belief that finetuning using meta-learning has to mimic the exact setting in the target task. A mismatch will hurt the performance [5]. However, in our study, we discovered that within the range of shots considered (from 1 to 4), the number of shots in finetuning did not significantly alter the accuracy. This observation underscores a key insight from our work: it is the total sample size $M \times m$, not just the number of shots, that determines the performance gains. This aligns with our theoretical findings (Theorem 3.2).

[5] Snell et al. Prototypical networks for few-shot learning. NeurIPS 2017.

Standard FT in Table 2 and All in Table 1

We illustrate “Standard FT” in the Section 4.3 Setup paragraph. We use multitask finetuning for results in Table 1 (including “All”) as illustrated in the Section 4 Experiment Protocol paragraph. Here we give a brief explanation.

In Table 2, "Standard FT'' refers to another finetuning method where an encoder is appended with a linear head that maps representations learned by encoders to the class logit probability, with the model finetuned based on logit loss. On the other hand, in multitask finetuning, the model is updated using the nearest centroid classifier. This approach differs from "Standard FT" in that it does not rely on the logits loss but rather uses class centroids computed from encoded samples. In Table 1, we use a multitask finetuning strategy and focus on comparing the task selection algorithm. "All" refers to results without a task selection algorithm and finetuning on all datasets.

We hope this explanation resolves the confusion and distinguishes between the two finetuning methods.

Typos of contrastive pretraining, cosine similarity, and missing index of theorem

We thank the identification of textual errors. We corrected these in the revised manuscript.

We thank the reviewer for the valuable comments and address the questions below.

Recent advancements in the field of multitask finetuning

We included the detailed related work in Appendix C. We refer the reviewer to Section Multitask Learning and Adapting Foundation Models in Appendix C for multitask finetuning. Please let us know if the reviewer has further concerns or suggestions.

Visual diagrams in Section 3.2

The procedure of the proposed task selection method was illustrated in Algorithm 1 of the main paper. Per the reviewer’s request, we further highlight the key intuition using the diagram in Figure 2 under Section 3.2 of the new revised manuscript.

​​Detailed ablation experiments for task selection

We conducted several ablation studies presented in Appendix G.4 that examine the task selection:

• G.4.1: Violate both consistency and diversity: We maintained the same level of sample complexity but incorporated data from unrelated tasks during finetuning. This resulted in a decline in performance.
• G.4.2: Only retain diversity while violating consistency: Introducing additional unrelated data during the finetuning phase also led to a deterioration in performance.

Per the reviewer’s request, we further included an additional ablation study on our task selection algorithm, focusing solely on either consistency or diversity, while compromising on the other:

• Ignoring Diversity: If the algorithm terminates early without fulfilling the stopping criteria, the data utilized in finetuning tasks fails to encompass all the attributes present in the target data. This leads to a breach of the diversity principle.
• Ignoring Consistency: Conversely, if the algorithm persists beyond the stopping criteria, the finetuning tasks become overly inclusive, incorporating an excessive amount of unrelated data, thus breaching the consistency.

We include details in Appendix G.4.4 and Table 9 of the new revised manuscript.

Descriptions for hyperparameter settings

We had detailed experimental settings with hyperparameters in Appendix G.2, including the number of tasks, the number of classes (N) per task, as well as the number of shot samples (K), and query samples (Q).

We have further expanded the appendix to include specifics on acquiring image embeddings and the methods used to measure consistency and diversity. Additionally, we have included information on the hyperparameters used during the finetuning optimization process. This expansion ensures that our experimental procedures are reproducible and that our findings can be validated. All these additional details are available in Appendix G.2.

Per reviewer AgCa’s request, we added an additional MAML baseline in Appendix G.8 and Table 14 for the second version of the revised manuscript. MAML can slightly outperform Adaptation and Standard FT under certain settings (DINOv2 ViT-S on tieredImageNet), but lags behind our multitask finetuning. The reason is in few-shot tasks, support samples (shot image) are very limited, preventing the model from learning a good set of parameters for adapting to new tasks.