Scalable Zeroth-Order Fine-Tuning for Extremely Large Language Models with Limited GPU Memory

We thank Reviewer SBMa for their positive feedback and for acknowledging the importance of memory-efficient LLM fine-tuning.

1. On FP16 vs FP32 performance (reviewer's question):

• We thank the reviewer for this insightful question and would like to clarify that our paper does not claim that the proposed framework performs poorly under FP32. The key performance comparisons are always made between ZO2 and MeZO under the same precision level.

• We acknowledge that presenting FP32 and FP16 results in the same table (Table 2) may have unintentionally led to confusion. We apologize for this oversight. In the camera-ready version, we plan to split the results into separate tables for FP32 and FP16 to avoid this ambiguity.

• Importantly, our textual discussion does not directly compare FP16 and FP32, because we believe such comparisons are less meaningful. Instead, we consistently focus on how ZO2 improves over MeZO under each respective precision configuration.

• That said, if one insists on comparing FP32 and FP16, FP16 naturally has significant advantages in both memory usage (due to lower precision) and speed (due to Tensor Core acceleration).

• Our framework supports both precisions and remains effective in either mode. We hope this clarifies the intended focus of our evaluation.

2. On applicability beyond the OPT family (point 2 of "Reasons To Reject"):

We thank the reviewer for raising this point. While our main experiments focused on OPT for consistency with MeZO, ZO2 can be used for any decoder-only transformer.

To demonstrate this, we conducted additional experiments on the Qwen3 model family—a recent, widely used decoder-only LLM series. Architecturally, models like LLaMA, Falcon, and Mistral share similar block structures and can likewise be partitioned for offloading, but we chose Qwen3 as a representative and up-to-date test case.

As shown below, ZO2 consistently reduces peak GPU memory usage across Qwen3 models, with savings increasing at larger scales:

3. On Figure 1 and Figure 5 (point 1 of "Reasons To Reject"): We thank the reviewer for this suggestion. In the camera-ready version, we will move Figure 5 into the main paper and revise or replace Figure 1 to better illustrate the system design.

4. On techniques mentioned only in the appendix (point 1 of "Reasons To Reject"): We agree and will move the key ablation details (e.g., parameter update and memory reuse) from the appendix into the main text in the camera-ready version.

5. On the significance of our novelty We appreciate the reviewer’s comment and would like to clarify that our work in fact introduces several complementary contributions across insight, method, and empirical validation. To better articulate the scope of these contributions, we summarize them below:

• Insight-level contribution: We observe that a core structural property of zeroth-order (ZO) optimization—its use of dual forward passes in the same direction—makes it fundamentally more compatible with CPU offloading than first-order (FO) training. Unlike FO methods, which require both forward and backward passes and thus transmit model parameters twice (once for each phase), ZO performs two forward passes using only a single parameter upload per block. Furthermore, because the two forward passes proceed sequentially in the same direction, ZO2 allows parameter communication to be effectively overlapped with computation. This structure enables our scheduler to be both simple and efficient, avoiding the complex interleaving logic typically required by FO offloading systems.

• Method and system design: (1) A lightweight dynamic scheduler using three CUDA streams that achieves high throughput with minimal control logic. (2) An RNG state manager that preserves gradient correctness under block-wise, asynchronous execution—a requirement unique to our offloading setting. (3) Several practical engineering optimizations, including block-local preallocated memory buffers, parameter update reordering to eliminate redundant transfers, and AMP-aware low-bit compression.

• Empirical results: (1) We enable fine-tuning of OPT-175B on a single 18GB GPU, to our knowledge, the first for ZO methods. (2) ZO2 achieves up to 10× GPU memory savings, no accuracy loss, and comparable or improved throughput across model sizes (Tables 1–3). (3) Our ablation study demonstrates that each proposed module—scheduler, memory reuse, update strategy—independently contributes to throughput gains.

We thank the reviewer for their detailed feedback and for recognizing the uniqueness of our asynchronous overlapping design using three CUDA streams (upload, compute, offload), as illustrated in Figures 1, 4, and 5 and implemented in the code. Below, we address the specific concerns and clarify the novel contributions of our work.

1. Clarifying our contributions (main concern): While we acknowledge that ZO2 does not introduce new optimization algorithms per se, we respectfully emphasize that the value of our work lies in co-designing an effective system architecture that exploits the structural advantages of ZO methods. We summarize our contributions below:

• Insight-level contribution: We observe that a core structural property of zeroth-order (ZO) optimization—its use of dual forward passes in the same direction—makes it fundamentally more compatible with CPU offloading than first-order (FO) training. Unlike FO methods, which require both forward and backward passes and thus transmit model parameters twice (once for each phase), ZO performs two forward passes using only a single parameter upload per block. Furthermore, because the two forward passes proceed sequentially in the same direction, ZO2 allows parameter communication to be effectively overlapped with computation. This structure enables our scheduler to be both simple and efficient, avoiding the complex interleaving logic typically required by FO offloading systems.

• Method and system design: (1) A lightweight dynamic scheduler using three CUDA streams that achieves high throughput with minimal control logic. (2) An RNG state manager that preserves gradient correctness under block-wise, asynchronous execution—a requirement unique to our offloading setting. (3) Several practical engineering optimizations, including block-local preallocated memory buffers, parameter update reordering to eliminate redundant transfers, and AMP-aware low-bit compression.

• Empirical results: (1) We enable fine-tuning of OPT-175B on a single 18GB GPU, to our knowledge, the first for ZO methods. (2) ZO2 achieves up to 10× GPU memory savings, no accuracy loss, and comparable or improved throughput across model sizes (Tables 1–3). (3) Our ablation study demonstrates that each proposed module—scheduler, memory reuse, update strategy—independently contributes to throughput gains.

2. On RNG state management (main concern): RNG state management is a necessary design element unique to ZO2’s offloading-aware execution model. MeZO applies Gaussian perturbations monolithically over the entire model, and thus requires only global seed control. In contrast, ZO2 performs block-wise CPU offloading, where each transformer block is independently uploaded, perturbed, dual-forwarded, and offloaded. Without precise RNG synchronization, the random perturbations used in parameter update and the dual forward passes would diverge across blocks, breaking the correctness of gradient estimation. Our RNG manager solves this by recording and restoring per-block RNG states, thereby ensuring correctness under asynchronous execution and enabling ZO2 to match MeZO’s accuracy.

3. On the absence of low-level profiling data ("minor but needed"): We agree that adding a low-level CUDA stream trace (e.g., from PyTorch Profiler or Nsight Systems) could enhance the presentation of our overlapping mechanism. We have collected such traces during development, which confirm the expected overlap between upload, compute, and offload streams. We have now made a representative Nsight Systems trace publicly available at the following anonymous link: [ https://anonymous.4open.science/r/zo2paper-D175 ] This trace confirms the correctness of our 3-stream overlap scheduling in practice. In the original submission, we chose not to include this trace due to space constraints, as we had already validated the effectiveness of our scheduler through the ablation study (Table 3). In that table, ZO2 achieves throughput nearly identical to MeZO, but removing the scheduler results in a significant throughput drop across all models. This clearly demonstrates the practical benefit of our overlap mechanism. We fully agree, however, that visualizing this via profiling is helpful and will include such analysis in the final version.

4. On the use of the term “fused” (point 1 of "need clarification / modification"): We thank the reviewer for this suggestion and agree that the term “fused” may cause confusion. We will revise the text in the camera-ready version to use “combined” or “scheduled together” to more accurately reflect our design.

5. On “reusable memory” (point 2 of "need clarification / modification"): We agree with the reviewer’s clarification and will update the terminology in the camera-ready version to refer to our strategy as a “preallocated block-local buffer” rather than “reusable memory,” to more precisely describe our implementation.

We thank the reviewer for their thoughtful and encouraging feedback, and for recognizing both the insight and system design behind ZO2, including the use of RNG, scheduler, and memory optimizations.

We appreciate the recognition of ZO2’s insights and system design. As suggested, we have further validated the generality of ZO2 on additional models beyond OPT, specifically on the Qwen3 model family. The following table shows the peak GPU memory usage (in MB) of ZO vs. ZO2 across Qwen3 model sizes. ZO2 consistently outperforms the baseline, with greater memory savings as model size increases:

Regarding the comment on unified memory systems (e.g., Apple M-series chips), we focused our design and experiments on heterogeneous architectures with discrete CPU and GPU memory, as these remain widely used in large-scale LLM training across both academia and industry.

That said, we very much appreciate the reviewer’s suggestion to consider unified memory systems, which indeed may reduce the need for explicit offloading due to their shared memory space. We agree this is a valuable direction for future work and may inspire further adaptation of ZO2-style techniques to such architectures.