Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures

Thank you for your time and thoughtful feedback on our manuscript. Below are our detailed responses to the weaknesses and questions you raised:

• Weaknesses:

1. In Figure 2, $k$ refers to the number of shots, consistent with the MeZO paper, rather than the $k$ used to denote the outer iteration number in Section 4.2. We apologize for the confusion caused by this reuse of notation.

   Regarding the number of training steps, we measure them as the number of gradient estimations, consistent with the MeZO approach.

   Regarding the learning rate, since MeZO and LOZO differ in their algorithmic structures, we tuned the optimal learning rate for each method separately. To address your concerns, we also conducted additional experiments to evaluate MeZO's performance with a learning rate of $2\times 10^{-7}$, which matches the learning rate used for LOZO on the RoBERTa-large model in the case $k=512$. The results are presented below.

   Task	SST-2	SST-5
   LOZO	94.1	53.0
   MeZO	92.4	40.1

2. Thank you for this insightful question!

   • Regarding the results in Table 18 of MeZO, unfortunately, we were unable to reproduce the exact results despite using the MeZO codebase and conducting a comprehensive grid search over hyperparameters. One possible explanation for the discrepancy is that, in the pre-trained RoBERTa-large model, a specific layer is randomly initialized rather than pre-trained, which may account for the differences. However, we would like to emphasize that, even when compared to the results reported in Table 18 of MeZO, our LOZO approach still achieves superior performance.

   • We do not know whether your intuition that "for small-size models like Roberta, MeZO-LoRA is reasonable to perform better" is correct or not. Generally speaking, we believe fewer parameters may result in over-fitting and hence hurt generalization, which may explain why MeZO-LoRA performs worse than MeZO. In fact, according to Table 8, we find both LoRA and MeZO-LoRA perform poorly in the $k=512$ case but perform well in the $k=16$ case. We hypothesize that, with $k=512$, the larger data volume requires more trainable parameters to better handle the increased complexity.

   Additionally, you pointed out that MeZO-LoRA has fewer trainable parameters, potentially leading to faster convergence as indicated by Equation (18) in the draft. However, Equation (18) pertains to the global convergence of $\min_X f(X)$, while LoRA only optimizes the adaptaors—namely, the low-rank adapters $A$ and $B$—and does not optimize the full parameter set $X$. In other words, MeZO-LoRA is not solving the same problem as MeZO (and LOZO), and hence the convergence rate in (18) cannot precisely reflect the convergence rate of MeZO-LoRA.

Thanks for the response from the authors, which solved most of my concerns. Generally, I think it's a good paper for improving the performance of zeroth-order optimization and I would like to increase my score to 8.

However, I don't agree with some of the points in the response letter, even though these disagree will not influence the conclusion of this paper:

One possible explanation for the discrepancy is that, in the pre-trained RoBERTa-large model, a specific layer is randomly initialized rather than pre-trained, which may account for the differences.

I think this paper follows the experiment setup in MEZO setup, which utilizes a prompted-based fine-tuning method. This means we are performing language modeling tasks during the fine-tuning process to solve the CLS problem, which use the language modeling head instead of the random initialized cls head.

In fact, according to Table 8, we find both LoRA and MeZO-LoRA perform poorly in the K= 16 case but perform well in the case. We hypothesize that, with k=512, the larger data volume requires more trainable parameters to better handle the increased complexity.

We can see from the first-order result from the original LoRA paper, which uses a setup of a full training dataset, but the result between LoRA and Full model FT is still the same. This means, that even with a full training dataset, there is no obvious over-fitting. So there may be other reasons here that need to be explored.

Again, I think these problems are not limited to the problem proposed in this paper, which will not influence the conclusion of this work. The method proposed achieves performance improvement with better time efficiency. Thus, I suggest the acceptance of this paper.

• We appreciate your thorough review and valuable feedback on our manuscript. Below, we address the weaknesses you pointed out.

1. Clarification of Memory Efficiency

   Thank you for your detailed feedback. We would like to address a key misunderstanding regarding the focus of our contributions. The primary contribution of our work is not to demonstrate the general memory efficiency of LOZO compared to first-order algorithms, as this aspect has already been established by MeZO. Nor is it to showcase memory efficiency over vanilla MeZO. Instead, our critical contributions are as follows:

   (1) Performance improvement over vanilla MeZO. LOZO leverages the low-rank structure in gradient estimation to improve the accuracy performance of MeZO while maintaining nearly the same (and often slightly lower) memory cost. This is evident in the results presented in Tables 1–3, where LOZO significantly outperforms MeZO in terms of accuracy, highlighting its effectiveness.

   (2) Substantial memory saving compared to momentum-based MeZO. By exploiting low-rank gradient estimation, we propose LOZO-M, a momentum variant with negligible additional memory overhead for low-rank momentum storage. In contrast, MeZO-M requires storing a full-rank momentum variable, resulting in significantly higher memory consumption. Specifically, as illustrated in Table 1, LOZO-M requires only 2.84 GB of memory, compared to 5.89 GB for MeZO-M (a nearly 52% reduction). Given that momentum techniques can substantially improve performance across various tasks (as evidenced in Tables 8 and 9), addressing momentum's memory overhead represents a practical and impactful contribution.

   We believe these contributions provide meaningful advancements over MeZO.

2. Extension to Additional Reasoning Tasks

   To evaluate LOZO's effectiveness to enhance LLM's high-level cognitive abilities, we conducted additional evaluations on the WinoGrande dataset which assesses the reasoning abilities. The experiments were conducted using various sizes of the LLaMA model. (Due to limited computational resources, we were unable to test the performance of gradient-based FT on LLaMA-70B.) The results, summarized in the table below, demonstrate that LOZO outperforms MeZO in most scenarios, highlighting its effectiveness in promoting high-level cognitive abilities.

   Model	LLaMA-7B	LLaMA-13B	LLaMA-70B
   LOZO	66.0	67.6	72.1
   MeZO	64.3	67.2	72.1
   FT-LoRA	70.9	76.6	50.4
   FT	64.4	73.3	-

• Below, we provide our detailed responses to the questions you raised.

1. Memory Comparison for OPT-13B and LLaMA-70B

   We provide a comparison of memory consumption for the OPT-13B and LLaMA-70B models, as shown below. The MultiRC dataset was evaluated on both OPT-13B and LLaMA-70B. Due to limited computational resources, we were unable to include exact memory cost evaluations for LLaMA-70B using gradient-based full fine-tuning (FT). However, our findings confirm that LOZO-M offers significant memory savings compared to MeZO-M (approximately 50% reduction) and other gradient-based methods, such as FT and FT-LoRA.

   Optimizer	OPT-13B (Memory)	LLaMA-70B (Memory)
   LOZO	26.9 GB	135.5 GB
   LOZO-M	27.3 GB	138.1 GB
   MeZO	27.3 GB	136.0 GB
   MeZO-M	52.1 GB	270.0 GB
   FT-LoRA	102.4 GB	187.2 GB
   FT	315.2 GB	> 640 GB

   We have included these results in Tables 10 and 12 of the revised manuscript.

2. Evaluation on Reasoning and Instruction Following Datasets

   We have evaluated the WinoGrande dataset on the LLaMA model, and additional tests on several other datasets are presented in Table 11 of the updated manuscript.

3. Convergence Speed Comparison

   In our original manuscript, Figure 3 provided a comparison of the convergence speed between LOZO and MeZO. In the revised version, we have expanded this analysis by adding two additional experiments to evaluate convergence speed across different datasets. Furthermore, we now include a comparison of the GPU wall-clock time for LOZO and MeZO. These new results are presented in Figure 5 in Appendix D.2 of the revised manuscript. These results demonstrate that the LOZO algorithm achieves faster convergence than the MeZO algorithm.

We hope these responses can address your concerns. If you have further questions, we would be happy to provide additional clarification.

Thank you for taking the time to review our manuscript and for providing valuable feedback once again.

In response to your concern, we conducted additional experiments on the MMLU benchmark. Due to time constraints, we focused on fine-tuning a single dataset and used the fine-tuned model for testing. The results are presented in the table below:

These results were obtained by fine-tuning the LLaMA-13B model on the WinoGrande dataset using different methods, followed by testing on the MMLU benchmark. For comparison, we also include the performance of the original pre-trained model as a baseline. The results clearly demonstrate that LOZO outperforms MeZO, suggesting that LOZO is more effective at learning and retaining information for complex tasks.

We hope these responses can address your concerns. If you have further questions, we would be delighted to address any additional questions or concerns you may have.

Thank you for taking the time to review our manuscript. We greatly appreciate your valuable feedback. Below, we provide our responses to your comments:

• Questions:

1. Ablation Study:

   Thank you for the question. We would like to clarify that our original manuscript already includes an ablation study on these hyperparameters, as detailed in Appendix C.3. Based on the results in Figure 4 and Table 7, we observed that a small $r$ (e.g., $r=2$) is sufficient to achieve high accuracy. However, increasing $r$ (e.g., $r=8$) does not lead to performance improvements and can sometimes degrade performance. Given that a larger $r$ introduces additional memory and computational overhead, we limited $r$ to a maximum of 8 in our experiments.

   Furthermore, regarding the hyperparameter $\nu$ (which you referred to as $N$. It appears we do not have a hyperparameter $N$, so we assume you are referring to the subspace period duration $\nu$. Please let us know if you meant a different hyperparameter), our ablation study in Figure 4 and Table 7 shows that a very small $\nu$ negatively impacts convergence. This is likely due to frequent subspace shifts causing abrupt model changes, which destabilize the training process. Conversely, while a larger $\nu$ has a less pronounced effect, it also slightly reduces the algorithm's performance. Based on these findings, we typically set $\nu$ to 50 or 100.

2. Additional Curves:

   We have added two new curves in Figure 5 of Appendix D.2 in the revised manuscript, corresponding to OPT-13B and OPT-30B on two additional datasets. To enable a more comprehensive comparison, we have also included a comparison of wall-clock time on GPUs between our proposed LOZO and the MeZO algorithm.

• Weaknesses:

  Weakness 3 has already been addressed in our response to your second question. Below, we provide response to Weaknesses 1 and 2:

  • Following your comments, we have included experiments on LLaMA models of different model sizes (7B, 13B, and 70B) on various tasks. The memory saving are shown in the following table (see also the newly added Table 12 in the revised manuscript). It is observed that LOZO can save significant memory compared to FT and FT-LoRA.

    Optimizer	LLaMA-7B	LLaMA-70B
    LOZO	14.1 GB	135.5 GB
    MeZO	14.3 GB	136.0 GB
    FT-LoRA	32.7 GB	187.2 GB
    FT	281.6 GB	640 + GB

  • The accuracy improvements of LOZO on LLaMA-7B across various tasks are summarized in the following table (The superior results achieved by ZO methods are highlighted in bold). Notably, we also evaluated LOZO's performance on the WinoGrande dataset, which assesses reasoning abilities. Due to time constraints, we were unable to test additional MATH and instruction-following tasks. For results on LLaMA-13B and LLaMA-70B, please refer to the newly added Table 11 in the revised manuscript.

    Task	SST-2	WiC	COPA	SQuAD	WinoGrande
    LOZO	94.8	57.2	85.0	90.3	66.0
    MeZO	91.6	56.3	86.0	90.0	64.3
    FT-LoRA	95.1	69.4	84.0	91.2	70.9
    FT	94.2	72.3	83.0	90.6	64.4

We hope these updates and responses adequately address your concerns. If you have further questions or need additional clarifications, we would be happy to provide them.

Thank you for taking the time to review our manuscript. We greatly appreciate your valuable feedback. Below, we provide our responses to your comments:

• Questions:

1. Memory Cost Comparison:

   We conduct new experiments to compare the memory consumption of the proposed LOZO and LOZO-M algorithms with FFT and FT-LoRA on OPT-13B across two datasets. The results are shown in the following table (see also the newly added Table 10 in the revised manuscript):

   Optimizer	RTE (Memory)	MultiRC (Memory)
   LOZO	27.0 GB	26.9 GB
   LOZO-M	27.4 GB	27.3 GB
   FT-LoRA	79.0 GB	102.4 GB
   FT	250.0 GB	315.2 GB

   It is observed that both LOZO and LOZO-M achieve significant memory savings compared to FT-LoRA and FT. Additionally, we evaluated the memory consumption of LOZO and FT on LLaMA models of varying scales on the MultiRC dataset. The results are provided below (see also the newly added Table 12 in our manuscript).

   Optimizer	LLaMA-7B	LLaMA-70B
   LOZO	14.1 GB	135.5 GB
   FT-LoRA	32.7 GB	187.2 GB
   FT	281.6 GB	640 + GB

   The above results also demonstate the memory efficiency compared to FT and FT-LoRA.

2. Experiments Beyond SuperGLUE:

   Thank you for your comment. We would like to clarify that our original manuscript already includes evaluations of our methods with the OPT model on additional datasets beyond the SuperGLUE benchmark, including SST-2, SQuAD, and DROP (see Tables 2 and 3). These results demonstrate the applicability of our approach to a broader range of tasks. Notably, SQuAD and DROP involve relatively long contexts, further showcasing the robustness of our methods. In our revised manuscript, we have highlighted the corresponding text in Section 5 to address your concerns.

   To further demonstrate the generalizability of our proposed algorithm across language models, we have conducted additional experiments on LLaMA with datasets beyond SuperGLUE. These results are presented below are for LLaMA-7B (see also the results for LLaMA-13B and LLaMA-70B in the newly-added Table 11 of the revised manuscript).

   Task	SST-2	WiC	COPA	SQuAD	WinoGrande
   LOZO	94.8	57.2	85.0	90.3	66.0
   MeZO	91.6	56.3	86.0	90.0	64.3
   FT-LoRA	95.1	69.4	84.0	91.2	70.9
   FT	94.2	72.3	83.0	90.6	64.4

   In addition, to evaluate the performance on datasets with long-contexts, we conduct a new experiment on the TREC dataset, which is part of the LongBench benchmark. This experiment evaluated the performance of LOZO against MeZO on RoBERTa-large and OPT-13B. The results are presented in the table below:

   Optimizer	RoBERTa-large (TREC)	OPT-13B (TREC)
   LOZO	77.9	63.2
   MeZO	62.4	25.4
   FT-LoRA	75.8	-
   FT	83.7	75.8

   Due to the incompatibility between FSDP and LoRA, we were unable to perform FT-LoRA on OPT-13B within the limits of our computational resources.

3. Scalability Across Model Sizes on Different Model Families:

   To evaluate scalability across different model types and sizes, we performed additional experiments on the LLaMA family, including LLaMA-7B, 13B, and 70B. Below, we present the results for LLaMA-7B. For the results on LLaMA-13B and LLaMA-70B, please refer to Table 11 in Appendix D.3 of the revised manuscript. These results demonstrate that LOZO performs well for even larger language models.

   Task	SST-2	WiC	COPA	SQuAD	WinoGrande
   LOZO	94.8	57.2	85.0	90.3	66.0
   MeZO	91.6	56.3	86.0	90.0	64.3
   FT-LoRA	95.1	69.4	84.0	91.2	70.9
   FT	94.2	72.3	83.0	90.6	64.4

• Weaknesses:

  Regarding the weaknesses you mentioned, we believe they align with the questions raised earlier.

Thank you again for your valuable feedback, and we hope our responses address your concerns. If you have further questions, we would be happy to provide additional clarification.