Hybrid Fine-Tuning of LLMs: Theoretical Insights on Generalized Smoothness and Convergence

Thank you for your detailed review comment and the constructive feedback.

1. Novelty: We would like to emphasis that our observation on using larger learning rate for the PEFT module is based on the theoretical understanding of the hybrid smoothness structure. From our best knowledge, we have not identified the existing literature showing the such hybrid smoothness. We hope the reviewer could kindly provide the related references, which could be an evidence that further validates our theoretical understanding.
2. Theoretical contributions: Our result presents $O(\epsilon^{-4} + \epsilon^{-2}/\delta)$ sample complexity with the probability $1-\delta$, which improves the existing sample complexity $O(\epsilon^{-4}/\delta)$ of [Li2024]. This improvement is due to our refined proof process with using tighter concentration inequality compared to their original proof. As an example, if we hope to achieve the desired complexity with the probability at least $1-\epsilon^2$, our derived upper bound indicates that it requires at most $O(\epsilon^{-4})$ data samples, which is much sharper than $O(\epsilon^{-6})$ derived from [Li2024].
3. Empirical results:

• Question 1: The point of random shuffling.

  We consider the reshuffling-type SGD because such epoch-wise optimizers are typically more common in machine learning practice

• Question 2: The memory and computation cost.

  We appreciate the suggestion of adding the memory and computation cost. We didn't track such cost and we plan to follow this suggestion in the future; the theoretical memory and computation cost are valid in our revised paper.

  Optimizer	Theoretical Memory	Asymptotical Memory
  FO-SGD (LLM)	$\sum_\ell \max\\\{ \|a_\ell \|, \|x_\ell\| \\\} + \|x\|$	$\sum_\ell \max\\\{ \|a_\ell\|, \|x_\ell\| \\\}$
  Vanilla ZO-SGD (LLM)	$\|x\|$	$\|x\|$
  FO-SGD (PEFT)	$\sum_\ell \max\\\{ \|b_\ell \|, \|y\|_\ell \\\} + \|y\|+\|x\|$	$\|x\|$
  Hybrid ZO-SGD (LLM+PEFT)	$\|y\|+\|x\|$	$\|x\|$

Thank you for your detailed review comment and the constructive feedback.

Question 1: Is it not feasible to train each module sequentially?

Yes, sequentially updating each module is an alternative approach. However, it will double the time cost for each batch of data since it requires two rounds of forward + backward passes.

Question 2: The accuracy rate of hybrid tuning decreases.

Adding the PEFT module to the base model introduces additional parameters; therefore, the model is more likely to be overfitting. The validation loss exactly validate this phenomenon. The reason why this phenomenon only appears to the hybrid tuning method is that the hybrid learning rate significantly accelerates the training procedure, which is also evidenced by the training loss of Figure 3.

Thanks for your comment. The combination is indeed a trivial trick; however, our main contribution also identifies the hybrid smoothness of its inherent structure, which presents the theoretical foundations on why it is more effective to set different learning rates.

Thank you for your review and feedback. We add the following point-to-point responses. Due to the limited time of rebuttal phase, we didn't add additional experiments. However, we sincerely hope the reviewer could value our novelty in the theoretical analysis as discussed below.

Difference from [Li2024]: We would like to highlight the main theoretical difference between our analysis and [Li2024]: Our result presents $O(\epsilon^{-4} + \epsilon^{-2}/\delta)$ sample complexity with the probability $1-\delta$, which improves the existing sample complexity $O(\epsilon^{-4}/\delta)$ of [Li2024]. This improvement is due to our refined proof process with using tighter concentration inequality compared to their original proof. As an example, if we hope to achieve the desired complexity with the probability at least $1-\epsilon^2$, our derived upper bound indicates that it requires at most $O(\epsilon^{-4})$ data samples, which is much sharper than $O(\epsilon^{-6})$ derived from [Li2024].

Lacks analysis or testing: We have provided theoretical analysis with the convergence guarantee and empirically verified the performance of our proposed method. We have clearly stated the methodology in our paper: Our method is simply applying different learning rates, which is motivated by the inherent hybrid structure of PEFT method.

Experiments are weak: We have tested our methods over three representative datasets: Test Classification, Question Answering, and Common Sense Reasoning. Adding more datasets in the same category won't enhance our statement.

Here, we answer the question raised by the reviewer:

• Question 1: Comparison of memory and wall-clock time.

  We appreciate the suggestion of adding the comparison of memory and wall-clock time. We have added a theoretical memory and wall-clock time of our proposed method as follows:

  • Memory Cost: We summarize the memory consumption comparison in the following table. The memory consumption of the PEFT model’s parameters is commonly significantly smaller than the base language model's parameters. Therefore, in the asymptotic analysis, we can omit its impact on our memory estimation.

    Optimizer	Theoretical Memory	Asymptotical Memory
    FO-SGD (LLM)	$\sum_\ell \max\\\{ \|a_\ell \|, \|x_\ell\| \\\} + \|x\|$	$\sum_\ell \max\\\{ \|a_\ell\|, \|x_\ell\| \\\}$
    Vanilla ZO-SGD (LLM)	$\|x\|$	$\|x\|$
    FO-SGD (PEFT)	$\sum_\ell \max\\\{ \|b_\ell \|, \|y\|_\ell \\\} + \|y\|+\|x\|$	$\|x\|$
    Hybrid ZO-SGD (LLM+PEFT)	$\|y\|+\|x\|$	$\|x\|$

    Where $a_\ell, b_\ell$ represents the total activations being stored for computing the backward gradients and $x_\ell, y_\ell$ represents the number of parameters in $\ell$-th layer of the base LLM model and the PEFT model, respectively.

  • Wall-clock time: Our method doesn't involve any modification on the original backward and forward step. Therefore, our wall-clock time is the same as the classical zeroth-order method reported in MeZo and Zo-Bench papers.

• Question 2 & 3 & 4: Follow the experiments given in MeZo & First-order SGD and Adam baseline & Revise Table 1 to add references.

  We appreciate this suggestion. We will add these experiments in the future.

• Question 5: Statement above Eq.(2) is incorrect.

  Thanks for pointing it out. We will fix this typo.

• Question 6: Optimizers used need to be explained.

  We use the vanilla SGD without momentum and weight decaying for both LLM and PEFT modules. We will make it more clear in our future submission.