Enhancing Zeroth-Order Fine-Tuning for LLMs via Gradient-Guided Subspace Selection

AW1: Limited performance gains and computational time analysis

Sorry for the confusion. Although the performance improvement by our method may appear modest, as shown in Table 2, it is significantly higher than the improvements obtained by other variants of ZO. We agree that our method does not always surpass other ZO techniques across various datasets. However, this observation can also be applied to other ZO methods, which also struggle to achieve optimal performance across all datasets or tasks. For example, although S-MeZO performs best on 4 out of 7 classification datasets, its overall classification performance is still worse than that of SVD-0. Meanwhile, SVD-0 performs significantly better than S-MeZO on both multiple-choice tasks and generation tasks. Note that, as shown in Table 2, our SVD-0 method yields the best performance in most datasets and task types, indicating the overall superiority and generalization ability of our approach.

We conducted new experiments to compare the computation time between our approach and two ZO methods (i.e., MeZO and SubZero). In the experiments, we fine-tuned each model for 20,000 steps on different datasets. The training time results (in minutes) are shown in the table below. The table shows that the training time for SVD-0 is slightly longer (by under 7%) than that of the two ZO variants. However, this additional time is minor compared to the improvements achieved in the classification, multiple choice, and generation tasks.

AW2: Excessive GPU memory usage

We conducted new experiments to investigate the memory usage for larger models (i.e., OPT-1.3B and OPT-13B), whose results are shown in the table below. From this table, we find that our method requires slightly more memory (less than 4%) compared to alternative techniques, mainly because SVD-0 must retain the $U$ and $V$ matrices, which are essential for updates in each layer.

AQ1: Why SVD superiority over alternative dimensionality reduction techniques

We conducted new experiments to understand why SVD outperforms other dimensionality reduction techniques. The following table compares our dimensionality reduction approach ("After SVD") with its counterparts (PCA, NMF, Factor Analysis, Random Projection, and t-SNE) on the calculated gradients. From this table, we can see that our approach achieves the highest mean value (0.4249) among all dimensionality reduction methods, while having a smaller standard deviation (0.0257), which demonstrates its stable and excellent performance.

AW1: Excessive GPU memory usage

We conducted new experiments to investigate the memory usage for larger models (i.e., OPT-1.3B and OPT-13B), whose results are shown in the table below. From this table, we find that our method requires slightly more memory (less than 4%) compared to alternative techniques, mainly because SVD-0 must retain the $U$ and $V$ matrices, which are essential for updates in each layer.

AW2: Theorem 1 violates the lower bound of SGD sample complexity

Sorry for the confusion. Our Theorem 1 does not violate the lower bound of SGD sample complexity. This is because our approach is based on zero-order SGD rather than first-order SGD. Please refer to the work [R1,R2] for a similar usage of the zero-order SGD in the convergence proof.

[R1] Yu et al., Zeroth-Order Fine-Tuning of LLMs in Random Subspaces, ICCV 2025 (arXiv:2410.08989).

[R2] Malladi et al., Fine-Tuning Language Models with Just Forward Passes, NIPS 2023.

AW3: Insufficient performance improvement in Table 2

Sorry for the confusion. Although the performance improvement by our method may appear modest, as shown in Table 2, it is significantly higher than the improvements obtained by other variants of ZO. We agree that our method does not always surpass other ZO techniques across various datasets. However, this observation can also be applied to other ZO methods, which also struggle to achieve optimal performance across all datasets or tasks. For example, although S-MeZO performs best on 4 out of 7 classification datasets, its overall classification performance is still worse than that of SVD-0. Meanwhile, SVD-0 performs significantly better than S-MeZO on both multiple-choice tasks and generation tasks. Note that, as shown in Table 2, our SVD-0 method yields the best performance in most datasets and task types, indicating the overall superiority and generalization ability of our approach.

AQ1: Memory usage

Please refer to AW1 for the memory usage of larger models using our approach.
As the experimental findings indicate, SVD-0 performs on par with MeZO or LoZO. In the future, we plan to investigate various memory optimization methods, which can further minimize the memory usage required by SVD-0.

AQ2: Complexity analysis clarification

Please refer to AW1 for the memory usage of larger models using our approach.

AW1: Performance inconsistency on different datasets

We agree that our method does not always surpass other ZO techniques across various datasets. However, this observation can also be applied to other ZO methods, which also struggle to achieve optimal performance across all datasets or tasks. For example, although S-MeZO performs best on 4 out of 7 classification datasets, its overall classification performance is still worse than that of SVD-0. Meanwhile, SVD-0 performs significantly better than S-MeZO on both multiple-choice tasks and generation tasks. Note that, as shown in Table 2, our SVD-0 method yields the best performance in most datasets and task types, indicating the overall superiority and generalization ability of our approach.

AW2: Unclear claims Please refer to the discussions in AQ1-AQ5.

AW3: Need for extended experiments on additional models

Thanks for your suggestion. We conducted new experiments based on the Qwen-1.8B model. Given the constrained time available for rebuttal, we exclusively compared our method against the baseline (MeZO) and the two most recent ZO baseline techniques (LOZO and SubZero). From the table below, we can find that our approach still achieves the best performance, indicating the adaptability and generalization ability of our method on models with different architectures.

AQ1: Explanation of cosine similarity results in Fig. 1

Sorry for the confusion. In Figure 1, we use the notation "Original" and "after SVD" to denote cases without and with the application of the SVD operation on calculated gradients, respectively.
From this figure, we can see that "after SVD" achieves higher cosine similarities, indicating that it can more accurately capture the gradient information from its estimated counterpart. To confirm this observation, we performed additional experiments employing various dimensionality reduction techniques. The results, which are presented in the table below, indicate that utilizing SVD significantly enhances our approach's ability to reconstruct gradients.

AQ2: Implementation details of GeneratePerturbMatrix(r)

Sorry for the confusion. We have defined GeneratePerturbMatrix(r) in Algorithm 1 and provided the implementation details in Section 4.1.

AQ3: Computational complexity comparison with SVD overhead

We agree that our approach can result in extra computation overhead due to the extra SVD operation. To understand the impact of the SVD operation, we conducted a new comparison between two representative ZO variants (i.e., MeZO and SubZero) and our SVD-0 method. The following table shows the results of the training time comparison (in minutes), indicating that SVD-0 needs slightly longer training time (by 7%) than the two variants of ZO. Note that, since the time complexity for SVD processes is O (n 3), where n represents the dimension of the matrix, the upper bound of the training time complexity remains unchanged. In practice, the additional time caused by SVD operations is minor compared to the improvements achieved in the classification, multiple-choice, and generation tasks.

In the future, we plan to explore various truncated SVD methods, which can reduce the computational overhead of our approach. For example, the dashSVD algorithm proposed in the paper "Algorithm 1043: Faster randomized SVD with dynamic shifts" can be integrated into our approach to accelerate its training process.

AQ4: Memory usage concerns

We conducted new experiments to investigate the memory usage for larger models (i.e., OPT-1.3B and OPT-13B), whose results are shown in the table below. From this table, we find that our method requires slightly more memory (less than 4%) compared to alternative techniques, mainly because SVD-0 must retain the $U$ and $V$ matrices, which are essential for updates in each layer.

AQ5: Practical determination of $L_1$ parameter in Theorem 1

Sorry for the confusion. Similar to other work (e.g., [R1]), the parameter $L_1$ in Theorem 1 is a constant that is used solely for convergence analysis. In practice, the selection of the step size can be done without needing any prior explicit knowledge of $L_1$.

[R1] Yu et al., Zeroth-Order Fine-Tuning of LLMs in Random Subspaces, ICCV 2025 (arXiv:2410.08989).

Dear Reviewer J5tk,

According to the conference policy, the reviewer must be involved in the author-reviewer discussion.

Could you please review the author's response and check whether they have addressed your concerns?

Thank you.

AC

AW1: Missing evaluation on cutting-edge models and modern tasks

Thanks for your suggestion. We conducted supplementary experiments to verify the generalization ability of the SVD-0 method on contemporary cutting-edge models and tasks. The results are as follows:

a) Performance Verification on Cutting-Edge Models

We conducted further experiments utilizing the Qwen-1.8B model, with the results detailed in the table below. These experiments indicate that SVD-0 also performs exceptionally well with the Qwen-1.8B model, clearly showcasing the versatility and generalization capability of our approach when applied to models of various architectures.

b) Evaluation on Diverse Task Types

We examined various datasets focused on financial sentiment analysis for evaluation purposes. The table below displays the experimental results, illustrating the consistent stability and effectiveness of the SVD-0 method across different domains and task types.

AQ1: Computational cost comparison with other ZO methods

Sorry for the confusion. We conducted new experiments to compare the computation costs between our approach and two ZO methods (i.e., MeZO and SubZero). In the experiments, we fine-tuned each model for 20,000 steps on different datasets. The training time results (in minutes) are shown in the table below. The table shows that the training time for SVD-0 is slightly longer (by under 7%) than that of the two ZO variants. However, this additional time is minor compared to the improvements achieved in the classification, multiple choice, and generation tasks.

AQ2: Testing on recent model architectures and contemporary tasks

We conducted new experiments on recent model architectures and contemporary tasks. Please refer to AW1 above for more details. We can find that our approach has the following merits:

a) Compatibility with Modern Model Architectures

Our approach is naturally independent of any specific architecture. The findings from the Qwen-1.8B experiments, outlined in AW1, demonstrate that the SVD-0 method effectively accommodates diverse model structures, including GPT-style models and instruction-tuned LLMs.

b) Generalization in Contemporary Scenarios

By conducting experiments across different tasks and datasets, we confirmed the efficacy of the SVD-0 method in modern applications. These experiments encompassed both classic NLP tasks and innovative areas, such as financial sentiment analysis, underscoring the method's broad range of applications.