GOODFIT: A Deep Learning Optimizer Fine Tuned for Fine Tuning

The experimental section needs to be revised. In particular, it should be extended to NLP, a domain in which the GOODFIT would be naturally applied.

Applicability issue in LLMs
Methods for finetuning LLMs currently rely on low-rank approximations of the weight matrices (LoRA) or through adapter heads. These are usually randomly initialized or zero-initialized (see e.g. LoRA [1]) and trained on top/aside from the pre-trained model. GOODFIT can not be applied as-is to state-of-the-art models for NLP since it requires starting from a converged weight matrix. Further validation of GOODFIT in one of those cases is crucial for the contribution of this work to be applicable exactly where it is most needed: in the case of LLMs.

Extension of the experimental validation
More generally, the experimental tests should be expanded. Currently, there are three tests: one on synthetic data, another on CIFAR100, and a third on the Waymo Open Motion Dataset. These tests are insightful and important, but only one is on a large-scale dataset, and none of those is done on natural images or textual data.
Since the cost for finetuning a model pre-trained on ImageNet or a large text corpus is relatively low, and the current method's impact could be significant in the NLP domain, I suggest adding experimental evidence in support of the GOODFIT performance on real-world image data and/or text.

Discussion of the limitation in the main text
The authors should have included a more thorough discussion of the limitations of their findings in the main text. The supplementary material contains a short discussion in which the authors address some of those, including the “slightly increased memory footprint.” These statements should be made more quantitative because they depend on the finetuning setup (full rank, LoRA, ...) and the type of architecture they are applied to (e.g. transformers vs CNNs).
Discussing the limitations of the current work in the main text strengthens the results of the paper while hiding them in the supplementary material does not.

[1] Hu et al., LoRA: Low-Rank Adaptation of Large Language Models

1. My primary concern is the insufficient empirical evaluation. Since the training dynamics of neural networks are still a black box, all new optimizers that claim to be effective must be thoroughly validated. However, in this paper, only results on toy datasets are reported, which cannot convince me at all.

2. The computational cost of GOODFIT is significant. Even with n_ref = 1, the cost is 2x as a regular optimizer, which may be unaffordable. And due to the extra costs, the baselines in the experiment part must also include finetuning results with 2x iterations, which are missing.

1): The paper could benefit from a more comprehensive discussion and comparison of training with regularization techniques. For instance, it should at least consider basic regularization methods like weight decay, which can significantly improve training efficiency with fewer hyperparameters.

2): The key point of the paper assumes that if the stored reference state corresponds to a good critical point of the original model, then $\Delta$ corresponds to a good gradient direction that restores the original critical point. However, as optimization progresses, there is no guarantee that $\Delta$ always corresponds to an optimal direction.

3): The paper lacks a rigorous theoretical proof related to generalization or convergence analysis. In the informal proof, what if the pretraining loss $L_{old}$ and fine-tuning loss $L_{new}$ are quite different? In such cases, the loss surface can become notably distinct.

4): The paper does not provide a clear discussion of fine-tuning efficiency. Fine-tuning speed is at least 2$\times$ slower than vanilla fine-tuning, and it requires storing reference states in GPU memory during fine-tuning. This might be challenging to generalize to large foundation models with billions of parameters.

5): Few ablation studies are included in the paper. For instance, there is no exploration of the effects of different configurations for the three main hyperparameters mentioned in Section 3.2.

6): Both training and validation curves are expected to be included to demonstrate the regularization effect. As shown in Table 1, head only fine-tuning outperforms the full model fine-tuning, indicating that the model is overfitting and many regularization techniques might work, questioning the necessity of GOODFIT.

7): The scalability and generalization capability of the proposed GOODFIT is not convincing. For example, how about using different optimizers, such as SGD variants, Adam, etc? Furthermore, experiments with large foundation models such as LLaMA and SAM would provide valuable insights.

8): The presentation can be improved. For example, the symbols in Figure 1 should align with the method descriptions.

• The hyperparameters of the reference methods are important to fully evaluate the strengths of the work, they should be described in more detail
• The connection to LwF a method which uses distillation seems a bit vague and is relied on in several places
• Related Work: The authors set out to tackle a very widely studied problem of finetuning with a vast literature. I was surprised to not see much mention of meta-learning methods which have been extensively used for “finetuning”. What is the connection to works such as
• Orthogonal Gradient Descent for Continual Learning
• Lookahead Optimizers
• Reptile and MAML
• Experiments - the experimental results are promising, but I would like to see better motivation for the datasets used. Transfer learning is widely studied and there are many standard datasets and fine tuning tasks results in the literature. The topic of this paper is a potentially high impact one but given the claims of providing a better generic finetuning method I believe the experimental results need to be more comprehensive and include datasets like e.g. VTAB
• Theoretical analysis: some of the assumptions seem to undermine the interest of the results