MeZO-A$^{3}$dam: Memory-efficient Zeroth-order Adam with Adaptivity Adjustments for Fine-tuning LLMs

• I question the proposed method's dimension-free claim, as the theorem appears problematic. Specifically, the proof of Lemma C.6 is incorrect, although I haven't verified all the theorem's proofs. There is a missing of the model dimension d. It is unknown whether the theorem can be proven to exclude d if this lemma includes d. See the same proof of Lemma 7 in the following paper:

Subzero: Random Subspace Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning

https://arxiv.org/pdf/2410.08989

• The paper should provide more practical verification of the proposed method, such as a setting for $\delta$ applicable to any model, rather than focusing heavily on theoretical analysis. The paper needs a method for setting $\delta$ across all models without requiring adjustments when model dimensions change. Currently, it necessitates different $\delta$ setting for each network. Also, no ablation experiment on $\delta$ has been conducted. The performance variations with different $\delta$ settings are not addressed. Additionally, the $\delta$ setting is influenced by the model dimension d, with no ambiguity regarding full parameter fine-tuning. However, it does not clarify whether the $\delta$ setting method should depend on the parameter dimension d or the dimension of the learnable parameters in parameter-efficient fine-tuning schemes, e.g., LoRA.

• The experimental results are weak, as the networks are small and single-task. LLM fine-tuning generally focuses on large models, such as model parameters exceeding 7B. The GPUs used in this paper are all advanced ones, which are fully capable of fine-tuning models above 7B. Why did not the authors do it? Since the paper relies on MeZO, it is a reasonable choice to repeat its experiments.

Though interesting, my major concern is if the same performance can be achieved by baselines by tuning $\alpha$, $\beta_1$ and $\beta_2$.

• Despite its superiority over other zeroth-order optimizer baselines, the widely used first-order baselines, e.g., SGD and Adam, are missing. This comparison is important for understanding the trade-offs between memory savings, performance loss, and real-world applications.
• The paper lacks comprehensive ablation studies on different components of their method. For example, they don't thoroughly explore different scaling functions h(d) for adaptivity adjustment, despite its importance to their method.

This paper has the following weaknesses:

• I would suggest the authors make the color system in the plottings consistent, as the conflicting colors in Figure 2, Figure 3(a) and Figure 3(b) is confusing at first sight.

• The authors constantly mention the "hand-crafted prompts" is a weakness of prior arts (Line 17, Line 66, Line 721). I am wondering does the method proposed in this work solve this problem? Did the authors mention it somewhere in the paper? I did not find the relevant discussion so I got a bit confused.

• The 8-bit trick can be used to all the methods equipped with Adam, right? I would suggest the authors report the performance/efficiency of 8-bit MeZO-Adam.

• In the MeZO paper, the authors discussed a lot of PEFT adds-on for ZO method, which can further save a lot of memory while maintaining high-level performance as well. I am wondering if the authors have tried similar experiments in this work? I guess the advantage brought by the adaptivity will be greatly hurt if PEFT modules are applied, right? This is because the parameter space $d$ is much smaller, if LoRA, for example, is used.