Simultaneous Computation and Memory Efficient Zeroth-Order Optimizer for Fine-Tuning Large Language Models

W1: Contribution lacks some novelty as components are borrowed from closely related work. For instance, LISA proposed by Pan et. al. 2024, uses a similar layer-wise sampling that is applied to first order optimizers like AdamW. The convergence analysis presented in the paper is also based on Sparse-MeZO by Liu et al. 2024, with minor changes.

W2: Although Sparse-MeZO was mentioned as a related work, it was not used as a baseline to evaluate against. Including this can be important to observe performance gaps between the two approaches, which both apply a form of parameter selection to improve MeZO.

• The theory is based on the assumption of Lipchitz continuity. In the context of deep learning and LLMs, this assumption can be overly simplistic or unrealistic. Loss landscapes in deep neural networks are often non-convex and may not satisfy Lipschitz continuity, not to say the billion params LLMs.

• The paper primarily compares LeZO with MeZO. Including comparisons with other state-of-the-art optimization techniques for fine-tuning LLMs, such as first-order methods (e.g., Adam, AdamW) or other memory-efficient optimizers, would provide a more comprehensive evaluation. (the LLM community are still generally using AdamW, I am not convinced if this appeals to adam users)

• The claim that LeZO achieves accelerated computation without additional memory overhead could be elaborated upon.

• The setting of LoRA could sway the performances by a lot with different rank and alpha, the paper only tested using r=8, and alpha=16, I would suggest try different LoRA set ups and offer evidence that it could work in the higher rank setting. Along the same vain, as the eval is done mainly by evaluating downstream performance, then adding more models into comparison would be more convincing.

1. Inconsistent Terminology and Notation: The paper suffers from inconsistent use of terms and symbols. For instance, the term "sparse rate $\rho$" on Line 194 and "sparsity rate reaches $\rho = 1$" on Line 451 seem to refer to entirely different concepts, which may confuse readers.
2. Errors and Ambiguities in Mathematical Notation: The mathematical expressions and symbols lack rigor and contain errors. According to Line 195, $\theta' \in \mathbb{R}^{\rho d}$ while $\theta \in \mathbb{R}^{d}$, making the addition of $\theta$ and $z'$ in Equation (4) undefined, as these vectors lie in different spaces. Additionally, cases where $\rho d$ is not an integer are not addressed. Notation in Lemma 1 is also quite unclear, making it difficult to interpret.
3. Errors in Tables: Table 1 contains unexplained elements, such as the appearance of "SSZO," which is not introduced or defined, potentially causing confusion.

1. The method consistently improves the convergence rate compared to MeZO; however, the rate still scales with d without further assumptions.
2. Building on point 1, the improvement in convergence rate is clear both theoretically and empirically. However, the corresponding impact on model performance remains unclear and unexplained.
3. There is a lack of empirical results on larger models, such as 30B. Additionally, testing on model types other than OPT should be considered.
4. Is the random selection of parameters optimal? Alternative selection methods, such as using importance sampling and weight norms, remain unexplored.