HFT: Half Fine-Tuning for Large Language Models

(1) The proposed method is too simple, too heuristic without novelties. Although the authors attempt to provide some theoretical explanations on page four, they are not convincing.

(2) Why HFT outperforms FFT? Because the number of trainable parameters of HFT is less than FFT, I think FFT could be deemed as an upper bound. In my understanding, the experiments are a trade-off of balancing downstream tasks and pre-training tasks. However, for many SFT-ed checkpoints, people would care more about the downstream abilities.

(3) This paper proposes a method to mitigate forgetting. In Section 4.2, the authors compare the method with some forgetting mitigation methods but there is one important and simple baseline missing: the model average, which is a strong baseline and should be included. Could you explain why model averaging was not included in the comparisons? It would be beneficial if you could incorporate this baseline into their experimental setup to provide a more comprehensive evaluation.

1. Lack of Distinct Advantage in PEFT Methodology
   • The paper proposes HFT as a new PEFT method but does not convincingly demonstrate its superiority over established alternatives like LoRA, Adapters, or BitFit. While the authors claim that "HFT allows LLMs to acquire new abilities while retaining and utilizing previously learned knowledge," this characteristic is inherent to all PEFT methods, not unique to HFT.
2. Positioning of the "Half-Reset" Technique
   • The paper introduces the "Half-Reset" approach, yet its relevance and novelty are unclear. The method is shown to be less effective than HFT and is not new; parameter merging is already well-studied in the literature, with "Half-Reset" representing a specific case of selective parameter merging (merging at a 0.5 ratio) [1]. This technique does not constitute a meaningful contribution of the paper.
3. Experimental Setup and Results
   • Unclear Benchmark Setting: The details of the TRACE benchmark setup lack clarity. It is unclear which tasks were used to evaluate continual learning baselines, and whether task IDs are accessible for training, given that methods like LoraSeqFT exhibit significant forgetting.
   • Lack of Baselines for Comparative Insight: To better evaluate the performance of continual learning methods, the paper should include non-continual learning baselines, such as pooling all data into a single learning process, training individual models per task, and then ensembling their performance.
   • Insufficient Comparative Value in Forgetting Reduction: The results only indicate that HFT reduces forgetting compared to FFT, which is not a surprising outcome, as HFT involves modifying fewer parameters during training. Other PEFT methods would likely yield similar results.
   • Parameter Selection Strategy in Table 4: The choice of selection strategies in Table 4 lacks intuitive appeal. Selecting 50% of parameters at the parameter level, rather than matrix-level, layer-level, or category-level, would be more straightforward and would precisely control the selection ratio.
4. Questionable Ease of Implementation Argument
   • The paper claims that "HFT does not change the model architecture, making it easy to implement and scale," but this does not set HFT apart from other PEFT methods like LoRA, which are already available in easy-to-use APIs and require minimal code for implementation and scaling.

[1] Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging.

1. Additional LLMs should be investigated to demonstrate the generalization ability of the proposed method.

2. In the continual learning setting, how does the proposed method compare to the baseline O-LoRA (Orthogonal Subspace Learning for Language Model Continual Learning)?

The method of freezing a portion of parameters during fine-tuning is relatively common, and numerous studies have already validated the effectiveness of this freezing method. As a result, the proposed method appears a little incremental and lacks sufficient innovation. Furthermore, the effectiveness of the proposed method in the paper largely stems from the fact that the dataset size of each sub-task is relatively small. Fine-tuning a portion of parameters makes the model less prone to overfitting, rather than truly addressing the issue of catastrophic forgetting. Additionally, for LLMs, the continual learning setting, which trains only one sub-task at a time, seems rather toy-like. Such a specialized scenario is difficult to apply in practical real-world situations.

Disadvantages: From the view of a new fine-tuning methods, the design of the method is trivial. Numerous existing studies [1-4] have explored partial optimization (parameter isolation) to improve model performance. The heuristic selection of 50% of freezing parameters does not provide a robust framework for further advancements in the parameter selection domain. The so-called "half fine-tuning" method appears to be a specific instance of freezing selection or parameter isolation. Selection criteria could be the closeness to convergence (measure the gradient norm change between two timestamp)[1], the long-term weight history[2], block-wise periodical update history[3] and importance based soft layer sampling strategies[4].

The authors provide only limited analysis of partial parameter fine-tuning. The paper lacks a detailed examination of the importance of each layer and the choice of optimizer. Additionally, there is an absence of optimization design for parameter selection and strategies based on the selected parameters. Authors could try weight norm[4], the gradient norm change between two training steps[1] and the layer-wise parameter difference to make measurements.

In continual learning, parameter isolation [5-8] have been long-standing techniques. It would be beneficial to understand the specific differences when these are applied to LLMs. Authors could tell their methods uniqueness from previous ones by clarifying its special design compared with them and list the difference exactly for each previous methods like DERPP[5], progressive networks[6]. According to the current version, I would regard HFT as a special case of the mentioned ones.

[1] AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning, arXiv 2021

[2] SmartFRZ: An Efficient Training Framework using Attention-Based Layer Freezing, ICLR 2023

[3] BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models, arXiv 2024

[4] LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning, NeurIPS

[5] Dark experience for general continual learning: a strong, simple baseline, NeurIPS 2020

[6] Progressive neural networks, arXiv 2016

[7] Pathnet: Evolution channels gradient descent in super neural networks, arXiv 2017

[8] Packnet: Adding multiple tasks to a single network by iterative pruning, CVPR 2018