Fine-tuning can cripple foundation models; preserving features may be the solution

Thank you for taking the time to review our paper and for your feedback! We provide our response below.

We would like to highlight that we design LDIFS to be an end-to-end fine-tuning method which also minimizes concept forgetting. Like all end-to-end fine-tuning methods, obtaining state-of-the-art accuracy on the downstream task at hand is a priority for LDIFS. This is a well-known problem with classic continual learning methods.

LDIFS is different from classic distillation.

• Most knowledge-distillation methods used in continual learning (CL) [A1, A2, A3, A4, A5] distill either from the logits of an expert model [A1, A3, A4] or from its last layer features [A2, A5].
• LDIFS distills not only from the last layer features but also earlier shallower features along the network architecture (see Appendix B for details). We find this modification to distillation to be crucial to LDIFS’s performance.
• To empirically see this, we compare with an ablation on LDIFS where we distill only from the last layer features. We show these results in Fig 11 of Appendix C.3 and find that using just last layer features is not nearly as performant as distilling from earlier features as well.

LDIFS outperforms distillation based CL and E2E fine-tuning on continual setups.

• We have improved our continual learning setup to 3 sequences of 3 tasks each and compare LDIFS with 5 different distillation based CL baselines. Results are in Table 3 of the revised draft. We find LP-init-LDIFS to consistently outperform all 5 CL baselines both on preventing forgetting and obtaining the best performance on the last task in the sequence, i.e., RESISC45.
• In addition, LDIFS minimizes concept forgetting compared to other E2E fine-tuning baselines as well, at the same time providing competitive accuracies on the last fine-tuned task. See Table 2 in the revised draft for results.

LDIFS’s simplicity makes it easily applicable to different architectures and foundation models.

• The LDIFS regularizer computes an L2 distance between the features of the pre-trained and current model during fine-tuning. Hence, it does not require any architectural changes to the network, which is what Bao et al. [1] does by training shallow classifiers with attention layers on intermediate features.
• This makes LDIFS easily scalable and applicable to larger architectures, as we show in Appendix C.7 (and Tables 8 and 9) where we study concept forgetting in CLIP RN50 and FLAVA ViT-B/16 and find LDIFS to be the most performant on these models as well.

Our results indicate that a network can encode learned concepts in activations of shallower layers as well, thereby making distillation from such layers necessary to minimize concept forgetting. We have explained this in Section 6 of the revised draft and updated our results. We hope that this explanation brings to light the novelty and benefits of the LDIFS regularizer compared to other distillation based CL/lifelong learning methods.

[A1] Li, Z. and Hoiem, D., 2017. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12), pp.2935-2947.

[A2] Jung, H., Ju, J., Jung, M. and Kim, J., 2018, April. Less-forgetful learning for domain expansion in deep neural networks. In AAAI 2018.

[A3] Rebuffi, S.A., Kolesnikov, A., Sperl, G. and Lampert, C.H., 2017. icarl: Incremental classifier and representation learning. In CVPR 2017

[A4] Hou, S., Pan, X., Loy, C.C., Wang, Z. and Lin, D., 2018. Lifelong learning via progressive distillation and retrospection. In ECCV 2018.

[A5] Zheng, Z., Ma, M., Wang, K., Qin, Z., Yue, X. and You, Y., 2023. Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models. ICCV 2023.

Thank you for taking the time to review our paper and for your feedback! We provide our response below.

“The experiments mainly compare with baselines with additional regularization to prevent forgetting.”

• To motivate the importance of end-to-end fine-tuning in foundation models, we compared with both prompt tuning and adapter baselines in Appendix C.4 and Table 5.
• In the revised draft, we have also provided comparisons with 5 continual learning baselines in Table 3 of the main paper.
• We have improved our continual learning setup from 2 sequences of 2 tasks to 3 sequences of 3 tasks each (see Section 5 of the revised draft).
• We have also incorporated Wise-FT as an ablation as we describe next.

“I think it will be better to add some experiments that compare the methods like WISE-FT or analyze whether the proposed method can combined with the WISE-FT”

Thank you for this suggestion.

• Wise-FT is a weight interpolation method which requires a pre-trained and fine-tuned model to work. Thus, it can be applied on every end-to-end fine-tuning baseline in our paper.
• In Appendix C.5 and Table 4 of the revised draft, we show the effect of applying Wise-FT on minimizing concept forgetting when fine-tuned on 3 tasks: CIFAR-10, EuroSAT and SVHN.
• We find that applying Wise-FT consistently reduces concept forgetting across all the fine-tuning baselines. However, the order of performance even after applying Wise-FT remains broadly consistent with LDIFS outperforming other baselines.

"It will be more clear if the results in Table 1 can be visualized in some other direct ways, e.g., take the car's value as the x-axis and others as the y-axis, then plot the performance of all methods on a picture."

• In Appendix C.6 and Figure 12 of the revised draft, we provide the visualization you ask for. For each downstream task and each baseline, we plot its test accuracy on the x-axis and the average test accuracy on all other tasks on the y-axis.
• We find LDIFS baselines to consistently lie around the top right corner of these plots indicating high accuracies both on the downstream task itself while preserving performance on other tasks.

“As we know, the hyperparameters of the regularization weights \lambda may affect the final performance of the fine-tuned model. I do not find any discussion about that. It will be better to add such kind of analysis about the hyperparameters.”

• In the previous draft, we did discuss the choice of $\lambda$ in the second paragraph of Appendix C.3, “Choosing $\lambda_{\mathrm{LDIFS}}$”.
• We perform a grid search over {0.01, 0.05, 0.1, 0.5, 1, 10, 100} and use the lambda which produces the highest accuracy on the validation set of the downstream task to be fine-tuned on. In our experiments, we consistently find the best $\lambda$ value to be 10 for all datasets, so all the LDIFS results we report are with $\lambda = 10$.
• In the revised draft, we have moved this paragraph to Appendix A (Training Details).

Thank you for taking the time to review our paper and for your feedback! We provide our response below.

“Additional discussion and comparison with continual learning methods would be necessary”

Thank you for this suggestion.

• In the revised draft, in Section 6 (Related Work), we provide a more detailed discussion of knowledge distillation in continual learning (CL), specifically highlighting how LDIFS differs from other distillation based CL methods.
• In Section 5, we have improved our continual learning setup from 2 sequences with 2 tasks each to 3 sequences with 3 tasks each, with Table 2 and Figure 6 reflecting these changes.
• In Table 3 of the revised main paper, we provide a comparison with 5 different distillation based continual learning methods on our improved 3-task sequence setup. We find LDIFS to consistently outperform all competitive continual learning baselines on all task sequences.

“Although the paper reported results of many experiments, I think the impact could be somewhat limited ... I think it would be interesting to do similar experiments in the paper to such other downstream tasks.”

Thank you for mentioning this interesting point. We will answer this in two parts.

Evaluating on ImageNet, a medium sized dataset with 1000 classes:

• First, to study concept forgetting and the effects of LDIFS on a medium sized dataset with a larger number of concepts, we include ImageNet (with 1000 classes) as the 10th downstream classification dataset in Table 1 of the revised draft.
• Since CIFAR-10/100 have classes within the set of ImageNet classes, we leave out CIFAR-10/100 when evaluating concept forgetting for models fine-tuned on ImageNet. As evident from Table 1, even with a 1000 class training dataset, our observations indicate both the existence of concept forgetting as well as the superiority of LP-init-LDIFS on minimizing concept forgetting.

Concept forgetting in CLIP beyond classification

• These is an interesting question, however, it is not obvious how we can quantify concept forgetting for CLIP when moving beyond classification to a task like object detection.
• The reason being that there is no straightforward way to quantify performance of a pre-trained CLIP model on object detection. If there were, zero-shot or linear probe object detection would be an immediate baseline in Owl-ViT [1], a work which specifically focuses on extending CLIP like models to open-vocabulary object detection.
• As far as recognizing concepts not seen in the fine-tuning set [1] is concerned, this observation is not very surprising. In our experiments, we observe a performance drop in classifying pre-trained concepts during fine-tuning and use this performance drop to quantify forgetting. In fact, the numbers in Table 1 indicate that the fine-tuned models are still able to recognize and classify pre-trained concepts, just not as well as the pre-trained models, thereby showing a certain level of forgetting.

We do think that this point is interesting and warrants more thought to design ways to properly quantify concept forgetting when moving beyond image classification. We look forward to investigating this as future work.

"Tables and figures are very small. Some of them could be merged (fig 2 and 5) or moved to the appendix."

Thank you for this point. We intentionally kept Figures 2 and 5 separate to maintain the flow of thought and not “spoil” LDIFS before we motivate LDIFS in our paper. We will work on increasing the figure size and readability in the final draft.

"Finetuning with reduced learning rate for the backbone [1] could be another baseline."

We tried the ablation of training with an initial learning rate of 1e-7 (100x lower than 1e-5) on the encoder for fine-tuning on CIFAR-10 (C10) and SVHN using all the fine-tuning baselines. We present the results in the table below where for each baseline, baseline+lr represents the ablation with lower learning rate on the encoder. The scores with and without this ablation are mostly competitive and we didn’t notice any particular trend in the results. Hence, we chose to leave it out of the revised draft.

Thank you for taking the time to review our paper and for your feedback! We provide our response below.

"While the phenomenon of “concept forgetting” is interesting, ... in some recent papers [a, b]."

"It will be great if the authors can highlight the conceptual innovation of the paper w.r.t. those missing prior works (e.g., [a-c])."

We explain the conceptual innovation of our work as follows.

Focus on E2E fine-tuning

• End-to-end (E2E) fine-tuning of foundation models is an active area of research and most works on this topic primarily focus on maximizing downstream task performance.
• Our paper is a take on i) how the latest E2E fine-tuning methods (like FLYP) applied to foundation models can lead to the forgetting of pre-trained concepts and ii) why minimizing concept forgetting should be an important additional criterion when designing fine-tuning methods, especially for foundation models which are a knowledge store of many pre-trained concepts.
• Unlike [a, b], we don’t start our investigation with well-known continual learning baselines, but with well-known E2E fine-tuning methods. Primary reason for this being that we do not want to devise an approach that compromises with the performance on the downstream task, a common observation among existing continual learning approaches.

A New E2E Fine-tuning method

• Catastrophic forgetting has been studied for a variety of models, and expecting concept forgetting in recent foundation models is not surprising. However, we not only talk about concept forgetting, we propose a simple way to quantify it, analyze all E2E fine-tuning methods on concept forgetting, and use insights from this analysis to propose a new E2E fine-tuning method, LDIFS, that minimizes concept forgetting without compromising downstream task performance.

Expanding to Continual Learning Setups

• Due to the applicability of LDIFS to continual setups, we explore its effectiveness in such settings as well (Section 5, Table 2) and also compare it with 5 other well-known continual learning baselines including LwF [c] and iCARL in Table 3 of the revised main paper. We find LDIFS to be consistently superior to all 5 CL baselines.
• Finally, the simplicity of LDIFS also allows it to be scaled and applied to different architectures and foundation models like CLIP RN50 and FLAVA ViT-B/16 (Appendix C.7).

Thus, our conceptual innovation compared to [a-c] lies in highlighting the importance of studying end-to-end fine-tuning methods from the lens of concept forgetting and bringing to light a simple and scalable fine-tuning approach that not only beats other fine-tuning methods but also outperforms well-known continual learning methods in minimizing forgetting.

"The proposed solution follows the knowledge distillation framework ... The innovation seems quite incremental."

• Unlike classic distillation methods which distill either from logits or last layer features, LDIFS uses earlier features along the network in the distillation process as well (details in Appendix B). This difference turns out to be crucial for LDIFS’s performance.
• We empirically show this in Fig 11 (Appendix C.3 last paragraph) of the appendix where we compare with an ablation on LDIFS where we distill only from the last layer features. This ablation significantly underperforms compared to the version of LDIFS using the full feature vector.
• As mentioned above, in Table 3 of the revised draft, we compare LDIFS with 5 distillation based continual learning baselines and find it to consistently outperform all these baselines on all the task sequences, thereby providing further empirical evidence on the importance of distilling from earlier features.

I appreciate the authors' effort in responding to my questions. However, I am not totally sure I am convinced by some of the responses.

Focusing on E2E fine-tuning of foundation models is definitely interesting, yet I am not convinced that this is one of the conceptual innovations. Foundation models are deep model pre-trained on large-scale data. Prior works has pointed that pre-trained deep models show catastrophic forgetting when fine-tuned on downstream tasks, and this issue needs to be addressed. That is why various methods have been developed (such as iCaRL and LwF) to mitigate the forgetting. If we still consider foundation models are a type of pre-trained deep models, doing the same things does not seem to conceptually novel to me.

I acknowledge that the paper presented a new E2E fine-tuning method. The question is that if the proposed method deems technically innovative. As mentioned in my previous comment, the proposed method is essentially feature-based knowledge distillation, which has been previously studied extensively. I shall refer the authors to Sec 2.2 in the survey paper [a] for many of the prior works that distill from intermediate features. Many of these feature-based methods has been considered for learning without forgetting (e.g., [b]). Again, I appreciate the empirical results yet have a hard time to consider this as a major contribution.

Many previous methods to mitigate catastrophic forgetting also talked about continual learning. Yes, the empirical results are promising, yet expanding the proposed method to continual learning, as a conceptual innovation, seems a bit weak.

[a] Gou, J., Yu, B., Maybank, S. J., & Tao, D. (2021). Knowledge distillation: A survey. International Journal of Computer Vision, 129, 1789-1819.

[b] Jang, Y., Lee, H., Hwang, S. J., & Shin, J. (2019, May). Learning what and where to transfer. In International conference on machine learning (pp. 3030-3039). PMLR.

Taking all these into consideration, I could not recommend this paper.