ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think

Q1: Caption of Figure 4

Thanks for pointing out! We've corrected it.

Q2: Missing Quotes and Descriptions

We've included quotes (EASE, CVPR-24 and APER, IJCV-24) and use one sentence to describe them.

Q3: Longer task sequences

As shown below, we evaluated the results on a task sequence of 20. ZO method still provides comparable performance across learning metrics. Besides, we will add evaluations on three other datasets for further comparison in the appendix.

Q4: Knowledge Transfer

Following [7], we provide the results of BWT and FWT of EASE with 20-split CIFAR-100 datasets above. We'll add these results and analysis into the Appendix of the main paper as an interesting plot. Thanks for your great suggestions to further enhance the quality of our manuscripts!

[7] A Comprehensive Survey of Continual Learning: Theory, Method and Application, TPAMI 2024.

Q5: Upper-bound for ZeroFlow

Thanks for your nice advice. We agree that the upper bound is significantly valuable for ZeroFlow. We added a linear layer for classification after the pretrained ViT-B/16-IN21K backbone and then fine-tune it with several ZO methods as the upper bound, as shown below. The updated results will be included in Table 1 of the revised manuscript. Moreover, we're opening up a leaderboard to the community, and will offer more upper bound results on various datasets and CL method. Thank you again!

Q6: Enhancement together

The results shown below indicate that the proposed enhancements do not conflict with each other and can be integrated. Specifically, FO training benefits from fine-tuning using hybrid ZO optimization, as illustrated in Figure 7. Additionally, it can be further enhanced by utilizing historical gradients and sparsity perturbation to reverse forgetting and achieve stable optimization, as analyzed in R2Q3. Thanks for your great suggestions.

Q7: Small Typoes

We've corrected the typo in Conclusion and added explanations for the bold terms in Table 1.

Q1: More Discussion of Enhancements

Indeed, as you note, Enhancement 3 has a potential acceleration advantage, which benefits from the reduction in average queries. More analysis see Enhancement 3 in our response 3 to Review HREV. Thank you for your suggestion, which effectively improves our proposed enhancement technique.

Q2: Visualization of Trajectory

We fully acknowledge that the small legend hindered the clarity of key insights. We will enhance figure readability by (i) enlarging all legends (old and new task), and (ii) zooming in on optimization endpoints (black, red and blue mark) and joint optimization centers (star mark). These revisions will ensure the visualized insights about task-specific optimization behaviors are more intuitively conveyed.

Q3: Concise Overview of ZO Series

Zeroth-order optimization aims to minimize/maximize an objective function $f: \mathbb{R}^n \to \mathbb{R}$ without derivative information. The core problem is formulated as $\min_{\theta \in \mathbb{R}^n} f(\theta)$, where $\theta$ denotes the optimization variable. To enable gradient-based updates, Simultaneous Perturbation Stochastic Approximation (SPSA) is a commonly used technique to approximate gradients by perturbing the input variables. Specifically, the gradient $\nabla f(\theta)$ at point $\theta$ is estimated as:

$\nabla L(\theta, \xi; B) = \frac{f(\theta + \epsilon \xi; B) - f(\theta - \epsilon \xi; B)}{2 \epsilon} \cdot \xi^{-1},$

where $\xi \sim \mathcal{N}(\mathcal{0}, \mathcal{I})$ is a random perturbation vector, and $\epsilon > 0$ is a small perturbation step size (typically adjusted during optimization).

ZO-SGD: Using the gradient estimator $\nabla L(\theta, \xi; B)$, zeroth-order algorithms, such as ZO-SGD, follow the iterative update rule:

$\theta_{t+1} = \theta_t - \alpha_t \nabla L(\theta_t, \xi_t; B),$

where $\alpha_t$ is the learning rate at step $t$. ZO-SGD bypasses explicit gradient computation through local function evaluations, making it suitable for high-dimensional, non-convex optimization problems.

ZO-SGD-Sign: A variant of ZO-SGD, known as ZO-SGD-Sign, improves upon the original approach by approximating the gradient direction using the sign of the gradient estimate. The update rule becomes:

$\theta_{t+1} = \theta_t - \alpha_t \, \text{sign}(\nabla L(\theta_t, \xi_t; B)),$

where $\text{sign}(\cdot)$ denotes the element-wise sign function. This approach often leads to faster convergence in some problems where the magnitude of the gradient is not as important as its direction.

ZO-SGD-Conserve: Another variant is ZO-SGD-Conserve, which conserves the gradient information over multiple iterations. The update rule for this method is:

$\theta_{t+1} = \theta_t - \alpha_t \cdot \frac{1}{k} \sum_{i=0}^{k-1} \nabla L(\theta_{t-i}, \xi_{t-i}; B),$

where $k$ is the number of past iterations used to average the gradient estimates. This method is beneficial when the gradient updates are noisy, and averaging helps stabilize the optimization process.

Forward Gradient: The forward gradient is an approximation technique where the gradient is computed by evaluating the function at a perturbed point and using a linear approximation. Specifically, the gradient $\nabla f(\theta)$ at point $\theta$ is estimated as:

$\nabla f(\theta) \approx \frac{f(\theta + \epsilon \xi) - f(\theta)}{\epsilon} \cdot \xi^{-1},$

where $\xi \sim \mathcal{N}(\mathcal{0}, \mathcal{I})$ is a random perturbation vector, and $\epsilon > 0$ is a small perturbation step size. The forward gradient is generally used when the optimization problem requires accurate directional updates without relying on an explicit derivative.

Q4: Extra Memory of Forward-Grad

Forward Gradient requires more memory than ZO-SGD because it involves computing gradients through the Jacobian-vector product, which necessitates storing all intermediate activations during the forward pass. In models like ViT, this includes storing large attention matrices and other intermediate results. In contrast, ZO-SGD only requires two forward passes with perturbed inputs, without needing to store these intermediate activations, resulting in lower memory usage.

Q1: Extensions to Contrib 1

We extended the experimental scope to enhance Contrib 1. In detail, we evaluated ZeroFlow on extra strategies: memory replay CL and VLM-CL (see Q5/7 of Reviewer see4).

Q2: Explanation of Contrib 2

In Section 3.2, ZeroFlow examines how ZO optimization helps mitigate catastrophic forgetting by comparing its optimization landscape to that of FO optimization. Figures 3 and 6 show the optimization paths of both methods as they balance preserving prior knowledge and adapting to new tasks. The axes represent the two most influential feature vectors, with the blue and red "X" markers indicating the optima for old and new tasks, respectively. The black dot represents the learned parameters, initially biased toward the old task, while the black star marks the optimal balance between both tasks. We will explain it in the revision.

Also, Reviewer Js2L also provided a clear explanation of Contrib 2 (Strengths and Designs Or Analyses Section) for your consideration.

Q3: More Discussion of Enhancements

Enhancement 1: Hybrid Optimization begins by leveraging gradients for fast adaptation to new tasks. Once the parameters are sufficiently close to the optimal solution, ZO is employed to fine-tune the solution, addressing forgetting.} i) FO ensures rapid convergence and efficient adaptation to new tasks, while ZO refines the solution in regions where gradients are unreliable or sparse. ii) The inherent randomness of ZO helps avoid sharp minima, akin to the principles of SAM, fostering more stable and generalizable solutions.

Enhancement 2: Historical Utilization employs an online Exponential Moving Average (EMA) to retain the past update information of old task gradients, adjusting them to minimize deviations from the historical direction.} By weighting the historical gradients, it reduces the impact of fluctuations induced by new tasks, effectively alleviating forgetting. Moreover, it enhances the stability of ZO optimization, ensuring smoother convergence and preserving knowledge from old and new tasks.

Enhancement 3: Sparse Perturbation introduces sparsity into the ZO by setting a fraction of perturbation dimensions to 0, thereby reducing the number of perturbed parameters.} i) This mitigates the instability inherent in ZO by lowering variance in gradient estimation, leading to more consistent updates. ii) Sparsity reduces overhead, making ZO method more practical for high-dimensional CL settings.

Overall, we will include above into the revision for clarity.

Q4: Effects of the Hyperparameters from CL

We provided the results on varying projection dimension $r$ and the trade-off parameter $\alpha$ of EASE. As below, ZeroFlow remain solid, which means that we provided reliable conclusions. We'll provide more analysis in version to ensure reliable conclusions. "‘Cons’ below refers to ZO-Conserve.

Q5: More Precise Statements

We've carefully reviewed the manuscript, clarifying unclear phrases, as follows,

• Gradient ban: We clarify that we've defined this concept in the original manuscript, see Lines 16, 54 and 40. And we've described its scenario to make the concept easier to understand, see Lines 23 to Lines 40.
• Forward pass method: this concept refer to gradient-free optimization methods that rely on forward pass, specifically referring to ZO and Forward-Grad method. We'll define it.
• Forgetting measure: The forgetting measure we used is a common metric in CL [5]. We will define it again for clarity.
• Function value: It represents the optimization objective in Figure 7, where values approaching zero indicate proximity to the global optimum.
• $x^{-1}$: denotes the element-wise inversion of the perturbation vector, which is necessary to ensure that the expected value of the gradient estimate aligns with the true gradient when $x$ follows a broader asymmetric distribution [6].
• Other misc: Small errors are fixed, e.g., the subtitle of section 3.2 corrected to ZO optimization for overcoming forgetting.

[5] A Comprehensive Survey of Continual Learning: Theory, Method and Application, TPAMI-24.

[6] Multivariate Stochastic Approximation using a Simultaneous Perturbation Gradient Approximation, TAC 1992.

Q1: Justification of Datasets

We follow the typical dataset setup to perform all evaluation [1,2]. In general, any overlap has been properly accounted for and domain gap is further considered (e.g. ImageNet-A and OmniBench are acknowledged to have large domain gap with ImageNet, please refer to [1,2]- Datasets Section). Thanks for your nice suggestion, we've stated it in revision for clarity.

[1] Continual Learning with Pre-trained Models: A Survey, IJCAI-24

[2] Class-Incremental Learning: A Survey, TPAMI-24

Q2: Concern about the Query Budget

Actually, ZeroFlow has comprehensively addressed this concern. In details, (i) Consistent Query Budget: ZeroFlow consistently operates under a query budget of 1 across all evaluations, which can be seen in Tables 1/2, Figures 1/2/4 etc. (ii) Enhanced Training Efficiency: The proposed enhancement methods retain the same query budget of 1 (see Figures 7/8, Tables 3/4.) while further accelerating training (see Enhancement 3 in our response 3 to Review HREV). (i) and (ii) strongly support our claim--Overcoming Forgetting is Easier. (iii) As you mentioned, the overhead analyses partly address this concern. Moreover, we provide extra insights on query budgets dynamics in Figure 5 to ensure a full understanding. We expect such an analysis to inspire subsequent work to extend the scope of ZeroFlow. Overall, this concern has been addressed in the original manuscripts. Your expertise in raising this issue is much appreciated.

Q3: Performance Discrepancy

Although both EASE and APER are prototype-based PTM CL models, EASE incorporates adapter training for each incremental task, wherein the training process builds upon previously trained adapters. In contrast, APER adapts the PTM solely during the initial training stage and subsequently maintains frozen for all following tasks. Consequently, the performance of later tasks in EASE is highly contingent on prior task training, whereas in APER, tasks remain independent of one another. Therefore, if a performance discrepancy arises between ZO and FO optimizers, EASE tends to amplify this gap, whereas APER is only marginally affected.

Q4: Open Discussion about Black-box LLM

First, our ZeroFlow naturally extends to LLM-as-a-service scenarios. Our tests on the VLM in the rebuttal (the results see Q6) have demonstrated this potential. Second, the forward-pass method we evaluate could be adapted to optimize prompts or lightweight external adapters through sequential API calls. In short, our findings on memory efficiency and forgetting mitigation potentially address key constraints of LLM deployment. Future work could establish benchmarks for black-box LLM continual learning.

Q5: Extra Eval. on Memory Replay Method

We further offer the performance of ZeroFlow on a typical replay-based method (MEMO [3], replay buffer=2000) to broaden applicability. As shown below, ZeroFlow remains stable in overcoming forgetting. Below, CFR and INA denote CIFAR-100 and ImageNet-A, respectively.

[3] A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning, ICLR-23 (Spotlight)

Q6: Potential Tests on VLM-CL

We evaluated ZeroFlow on continual learning of vision-language model (using MoE4Adapter [4], all training protocols follow [4]). As below, the general utility of ZeroFlow is reconfirmed. Furthermore, we are preparing to open up a leaderboard to the community that will cover more tasks, including PTM-CL, VLM-CL etc.

[4] Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters, CVPR-24.

Q7: Memory/Time Advantages on Larger Transformers

Yes, we had employed two larger transformers (ViT-L/16 and ViT-H/14) to evaluate the efficiency of ZeroFlow, as shown below. All of them offer memory advantages, and ZO with ZO-Sign still run faster than FO.