Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning

Thanks for your constructive comments, we carefully address your concerns below.

w1: Since the parameter selection is based on the smallest weights, the benchmark could be strengthened by including a comparison with the opposite strategy: selecting and updating only the largest-magnitude weights.

Thank you very much for this valuable suggestion.

Our focus on small-magnitude weights is based on their fundamental properties in pre-trained models and zeroth-order optimization.

Ablation Study: Our experimental analysis examines how weights of different magnitudes affect performance. We divided weights into 4 groups by magnitude, where '0-0.25' represents the bottom 25% and '0.75-1.0' represents the top 25%. The results in the tables demonstrate that updating small-magnitude weights consistently achieves better performance than updating large-magnitude weights.

Building on these findings, we performed additional experiments to analyze weight selection strategies. While updating small-magnitude weights (0-0.25) shows strong performance, we tested incrementally including more weight groups (e.g., expanding to 0-0.5, which includes both 0-0.25 and 0.25-0.5). The results show that adding weights beyond the small-magnitude group (0-0.25) actually reduces performance, confirming that focusing on small-magnitude weights is optimal.

w2: A potential area for strengthening the experimental comparison would be to include another recent and relevant method in sparse ZO optimization. For instance, comparing S-MeZO against SensZOQ would provide a more comprehensive picture of its performance within this specific sub-field.

We thank the reviewer for this excellent suggestion. SensZOQ is indeed a highly relevant sparse zeroth-order fine-tuning method, however, we didn't compare it in our original submissions as it published on arxiv several months after our work. Therefore, we believe it does not diminish the novelty and contribution of our paper. However, we totally agree that for completeness we should include a comparison and discussion in our paper.

Before presenting our comparison, we note key methodological differences: (1) SensZOQ requires additional computational cost to calculate first-order gradients for obtaining sparse masks, while our method dynamically generates masks without gradient computation; (2) SensZOQ targets extreme sparsity combined with quantization for edge devices, whereas we focus on balancing sparsity with performance for general memory-efficient training. Our experimental comparison on challenging tasks shows:

L1: Addressing the Need for Broader Discussion of Societal Impact and Dual-Use Considerations.

We thank the reviewer for this thoughtful suggestion about expanding our discussion of broader societal impacts. We agree that acknowledging the dual-use nature of making LLM fine-tuning more accessible is important and standard practice in LLM research.

We will expand the limitations section in the revised manuscript to include a discussion of broader societal implications. Specifically, we will acknowledge that while our method democratizes access to LLM fine-tuning through reduced computational requirements—enabling beneficial applications for researchers and practitioners with limited resources—it also potentially lowers barriers for malicious actors who might use fine-tuned models for generating disinformation or other harmful content.

We will also briefly discuss potential mitigation strategies, such as the importance of responsible deployment practices, monitoring fine-tuned model outputs, and the continued development of detection methods for AI-generated harmful content.

C1: The manuscript contains spelling and wording mistakes that should be fixed before resubmission.

We sincerely apologize for the spelling errors and will conduct thorough proofreading to correct all spelling mistakes and grammatical errors before resubmission.

We sincerely thank the reviewer sJB2 for the further feedback. Our responses are as follows:

W2: The comparison with SensZOQ is very helpful. May I kindly ask under which density or sparsity ratio these experiments were conducted? In any case, I truly appreciate the authors’ efforts—the results appear solid.

Thank you very much for your follow-up question about W2. Since the SensZOQ code was not publicly available at the time of our experiments, we followed the settings described in the SensZOQ paper and reproduced its performance. We conducted a grid search over parameter selection ratios, testing with {0.1%, 1%, 10%} of parameters selected for updates. We found that updating only 1% of parameters achieved the best performance on RTE and BoolQ, so we used 1% parameter density (i.e., selecting 1% of parameters for optimization) for all experiments in W2. The experimental results for determining the density are shown in the table:

Question 1: Zeroth-order optimization relies on the expectation that its gradient estimator converges to the true backpropagation gradient as the number of samples increases. If we increase the number of ZO samples to reduce noise, would updating small weights still outperform updating large weights?

Thank you very much for your constructive comment. We conducted experiments with larger batch sizes for Sparse MeZO (increasing batch size from 16 to 64) and compared the performance between updating only small weights versus large weights. We find that updating small weights still achieves better performance than updating large weights even with larger batch sizes.

The results are shown in the table below, where '0-0.25' represents the bottom 25% and '0.75-1.0' represents the top 25%. We observe that updating smaller parameters (0-0.25) with batch size 64 achieves higher performance than updating larger parameters (0.75-1.0) on both RTE (83.7 vs 60.3) and BoolQ (83.3 vs 67.6). We will provide the complete results in the revision.

Question 2: Moreover, following the authors' argument, does this imply that even in backpropagation (noise-free), restricting updates to small-weight or small-gradient parameters would be preferable? If not, doesn’t this suggest that the observed benefit is simply a workaround for ZO noise rather than an intrinsic optimization principle?

Thank you very much for your constructive question. We investigated this question by conducting experiments with first-order methods (Adam) on LLaMA-7b/RTE, selectively updating only small-magnitude weights vs. large-magnitude weights. Results show that first-order methods achieve similar performance regardless of weight magnitude selection (83.6% vs 83.1%, only 0.5% gap), unlike zeroth-order methods where small weights significantly outperform large weights (82.7% vs 57.3%). This confirms that the disproportionate weight magnitude impact is specific to zeroth-order optimization due to gradient estimation noise, rather than a general fine-tuning phenomenon.

Q3: Regarding the new observation: When training on top 20% of the largest weight vs the bottom 20%, how are the step sizes determined for each case and what are the impacts on the stability of the training and the performance of the model?

For step size determination, we conducted grid search over {1e−6, 2e−6, 3e−6, 4e−6, 5e−6} for both top 20% largest and bottom 20% smallest weights, selecting the best-performing learning rate for each configuration through validation performance. Both achieved optimal results at similar rates (2e−6 to 3e−6), ensuring fair comparison. Regarding training stability, small weights consistently converged across all tested learning rates with low variance, while large weights exhibited training instability and higher variance, particularly at rates ≥4e−6. The 15-20% performance gap persisted across different learning rates, confirming this is a fundamental property rather than hyperparameter sensitivity.

Q4: Is the disproportionate impact of the weights also observed when we train the model by first order method instead of zeroth order method?

We investigated this excellent question by conducting experiments with first-order methods (Adam) on LLaMA-7b/RTE, selectively updating only small-magnitude weights vs. large-magnitude weights. Results show that first-order methods achieve similar performance regardless of weight magnitude selection (83.1% vs 83.6%, only 0.5% gap), unlike zeroth-order methods where small weights significantly outperform large weights (82.7% vs 57.3%). This confirms that the disproportionate weight magnitude impact is specific to zeroth-order optimization due to gradient estimation noise, rather than a general fine-tuning phenomenon.

Thanks for your constructive feedback, we carefully address your concerns below.

w1: Some of the datasets are missing from some tables (for example, table 2 and others in the appendix). Please also add other baselines such as ZO-SGD to table 3 and where it’s relevant.

We thank the reviewer for this valuable feedback. We acknowledge the incomplete dataset coverage in some tables and missing baselines.

Missing datasets: Some results were omitted due to computational constraints, but we will complete these experiments for the revised manuscript.

Complete results of table 2:

Complete results of table 3:

We will provide more comprehensive experimental results in the revision with consistent baseline coverage across all tables.

w2: The algorithm introduces a sparsity parameter and it’s unclear how to set this parameter other than tuning

We thank the reviewer for this valuable observation. We determine thresholds using a principled sparsity-based approach. Specifically, we use a percentile-based method where the threshold is set based on a target sparsity level. For example, with 80% sparsity, we sort the weight values of each layer and set the threshold at the 80th percentile. Importantly, this threshold is determined once before training begins and remains fixed throughout the optimization process.

Our empirical results in Figure 4 demonstrate that a sparsity level of 75% consistently yields strong performance across diverse tasks (RTE, BoolQ, and WIC), highlighting the generalizability of our threshold selection method. Furthermore, we observe robust performance gains compared to vanilla MeZO across a range of sparsity values (70-80%), indicating that our method is not sensitive to the exact threshold choice. This stability across different tasks eliminates the need for task-specific threshold tuning.

w3: As acknowledged by the authors, there is still a significant gap between this method and gradient methods

We acknowledge the performance gap between our zeroth-order method and first-order gradient methods, which is an inherent limitation of ZO optimization.

However, we observe this gap is narrowing as pre-trained models improve. For example, the average performance gap between S-MeZO and Full FT decreases from 2.6% on LLaMA-7B to 1.2% on LLaMA-30B. This suggests that the performance gap between zeroth-order and first-order methods can be narrowed with increasing model capability.

w4: The theorem in the appendix only applies for a fixed masking, not the masking by threshold used in the algorithm.

We thank the reviewer for this important clarification. The reviewer is correct that our theorem assumes a fixed mask while our algorithm uses dynamic threshold-based masking.

However, our theoretical analysis provides a meaningful upper bound. Although the mask is dynamic, the number of selected parameters ($\hat{d}$) continuously decreases during training as parameters grow and exceed thresholds. Since our convergence rate depends on $\hat{d}$, and $\hat{d}$ monotonically decreases in our approach, the fixed-mask analysis serves as an upper bound on convergence rate.

This means our method may converge faster than the theoretical prediction. We will clarify this relationship in the revised manuscript to better explain how the theoretical analysis relates to our dynamic masking approach.

We sincerely thank the reviewer Waaa for the further feedback. Our responses are as follows:

Q1: Could you please check the numbers in your response to w3?

We apologize for the confusion. We have updated our results in the previous response. Additionally, to further explain how the gap (Full FT - Sparse MeZO) narrows as pre-trained models improve (from LLaMA-7B to LLaMA-30B), we provide the results on LLaMA-7B and LLaMA-30B in the table:

From the table, we observe that the average performance gap (the last column) between S-MeZO and Full FT decreases from 2.6% on LLaMA-7B to 1.2% on LLaMA-30B. This suggests that the performance gap between zeroth-order and first-order methods can be narrowed with increasing model capability.

In addition, we find that increasing the batch size can further reduce the performance gap. For instance, when we increased the batch size from 16 to 32 (with seed 0), the performance gap decreased from 1.9 to 1.3 on the RTE and BoolQ tasks.

Q2: I'm not sure I understand the response to w4. What do the authors mean by "the number of selected parameters continuously decreases" (why is it the case if the mask is percentile based)?

We apologize for the confusion. To clarify our setup: we determine the threshold for each layer once before training based on the target sparsity (e.g., smallest 20% of parameters), and this threshold remains fixed throughout training to avoid expensive recomputation at each iteration.

The number of selected parameters decreases because as training progresses, some parameters initially below the threshold will grow larger and become masked. Once masked, these parameters stop updating and cannot return below the threshold. Since the threshold is fixed, parameters can only transition from active to masked (not vice versa), leading to a monotonically decreasing active parameter count.

Thanks for your constructive and inspiring feedback, we carefully address your concerns below.

w1: The paper lacks discussion and comparison with some important ZO works in sparsity, like [1]. Moreover, seems the method works due to gradient estimation with less noise, so it’s helpful to compare with some variance-reduction works of ZO, like [2] and [3].

We thank the reviewer for pointing out these important related works in ZO optimization with sparsity and variance reduction. We agree that comparisons with these methods would strengthen our evaluation. We have conducted experiments comparing our method with SensZOQ [1] and ZO-SVRG [2] on LLaMA2-7B. The results show that Sparse MeZO consistently outperforms these baselines:

In addition, although variance reduction methods such as ZO-SVRG can improve gradient estimation, they typically introduce additional memory overhead by storing multiple model weights for variance reduction, which conflicts with our goal of memory efficiency. Our approach achieves variance reduction through selective parameter optimization without extra memory cost. We are conducting experiments with [3] and will include these comparisons in the final revision.

w2: There are no specific numbers regarding the computational overhead introduced by threshold calculation, and dynamic masking and perturbation layer by layer in the forward pass.

Thank you for your constructive comment about computational overhead. We have conducted detailed measurements to quantify the overhead and the results is shown in the folowing table. Due to the main overhead is from the definition of threshold and mask calculation, we mainly analyze them in the table:

(1) Threshold Calculation (MeZO-Threshold row in the table): The threshold determination process takes approximately 0.01 seconds for a LLaMA-7b model. In addition, it is a one-time cost before training begins, and therefore this additional computation is very limited compared with several hours training of each experiment.

(2) Dynamic Masking Overhead (MeZO-Mask row in the table): We provide the detailed dynamic masking calculation time and each training step time in the table. From this table, we can find that the masking operation adds approximately 0.07s per forward pass, and the training step time in these experiments varies from 0.8588s to 6.2621s. Therefore, the relative overhead is between 0.009 to 0.08, which is negligible.

w3: The overall contribution is incremental as the work adds common magnitude-based sparse steps on top of the MeZO optimization algorithm.

We appreciate this feedback and would like to highlight several substantial contributions of our work beyond MeZO:

Technical Innovation: We identified and addressed a fundamental limitation in ZO optimization: gradient noise impacts large weights more severely than small weights. This insight led to our selective parameter updating approach.

Performance Gains: Our method achieves substantial improvements over MeZO. For example, (1) 9% absolute accuracy improvement on RTE, (2) 3.5x faster convergence, (3) Comparable performance to full fine-tuning while maintaining MeZO's memory efficiency.

Novel Implementation: We developed a memory-optimized implementation for sparse masking that maintains inference-level memory consumption, enabling fine-tuning of LLaMA-30b on a single A100 GPU.

Thanks for your constructive feedback, below are our responses regarding your concerns.

w1: It seems possible to me that a gradient computed on one set of 16 points aligns with a gradient computed on another, completely independent set of 16 points. I would like to know if similar results would hold for supervised fine-tuning (SFT). In particular, to include a comparison in Figure 2(b) with SFT.

Thank you for this insightful suggestion. We agree that a more rigorous comparison is needed. Our new analysis compares SGD and MeZO using the same batch. The results show MeZO has a higher probability of loss increment (0.4-0.6) compared to SGD (0.2-0.3). While both methods can increase loss on different batches, MeZO exhibits this behavior more frequently.

w2: The results regarding the convergence rate in Section 4.4 and Figure 3 only compare convergence curves for MeZO and Sparse-MeZO, but it would be relevant to add the comparison with SFT here.

We thank the reviewer for this suggestion. Adding SFT convergence comparison would indeed provide valuable context for understanding the convergence characteristics of different optimization approaches. In our experiments, SFT typically converges within 1,000 steps, while both MeZO and Sparse-MeZO require 10,000 steps due to the inherent nature of zeroth-order optimization, which relies on gradient estimation rather than exact gradients.

We will add SFT convergence curves to Figure 3 in the revision. This comparison will help readers better understand the trade-offs: while SFT achieves faster convergence, it requires significantly more memory (12× more than our method). Sparse-MeZO provides a middle ground: faster convergence than vanilla MeZO while maintaining the memory efficiency of zeroth-order methods.

Q1: “Impressive results” line 1 seems like a superlative statement

We agree that the phrase "impressive results" contains subjective language. We will revise this to use more objective language in the revision. Specifically, we will change the sentence to: "While fine-tuning large language models (LLMs) for specific tasks often yields strong performance..."

Q2: Line 165, how are the percentages chosen to define small and large values? Why set the threshold to 20%?

We thank the reviewer for this valuable observation. For threshold, we use a percentile-based method where the threshold is set based on a target sparsity level. For example, with 80% sparsity, we sort the weight values of each layer and set the threshold at the 80th percentile. Importantly, this threshold is determined once before training begins and remains fixed throughout the optimization process.

Our empirical results in Table 12 demonstrate that a sparsity level of 70-80% consistently yields strong performance across diverse tasks (RTE, BoolQ, and WIC), highlighting the generalizability of our threshold selection method.

Q3: Lines 222-225, I am not sure how this paragraph helps with the argumentation. According to the author, this method is not aligned with the goal of the paper and is not discussed anywhere else in the paper.

We thank the reviewer for this feedback and agree with the suggestion. The paragraph discussing 1-bit quantization introduces an alternative approach that we do not pursue or evaluate, which may indeed cause confusion for readers. We will remove this paragraph in the revision for better clarity.

Q4: It would be helpful to report the standard variations on top of the mean across the three runs for the results in at least Table 1. This could help capture the stability of the approach.

We thank the reviewer for this valuable suggestion. We agree that reporting standard deviations would better demonstrate the stability of our approach across multiple runs.

We will update Table 1 in the revision to include standard deviations for all methods across the three random seeds. For reference, we have already provided standard deviations in Table 10 (Appendix) for our main LLaMA-7b results, which show that Sparse-MeZO maintains stable performance with standard deviations typically ranging from ±0.3 to ±1.6 across different tasks.

I thank the authors for their responses and clarifications.

• I maintain my reservations regarding weakness w1, particularly concerning the motivation, which I believe could still be refined. I encourage the authors to incorporate the new experimental results into the paper to strengthen this aspect.
• Regarding w2, I appreciate the quantitative details provided, which offer meaningful points of comparison.

Thank you as well for addressing my question — I am satisfied with the response.