Enhanced Gradient Aligned Continual Learning via Pareto Optimization

• The proposed methods are very computationally costly. The main algorithm is a nested algorithm consists of two algorithms. Furthermore, it is unclear about the complexity of the Frank-Wolf algorithm applied to continual learning. It would be better to provide some complexity analysis and running time comparisons.

• The proposed method is similar to weighting the different tasks in multi-task learning. The authors think that “However, these Pareto optimization based MTL methods have not been fully investigated in the context of dynamic streaming tasks and are therefore unsuitable for CL tasks.” The reviewer does not agree with this viewpoint. If we view the memory buffer data as multi-task data where each task takes some proportion, then the pareto optimal optimization in multi-task learning can be directly applied to continual learning. The proposed method needs to compare with those state-of-art multi-task learning methods with multi-objective optimization, e.g. [2]. The novelty of the proposed method is not high from this viewpoint.

• Applying multi-objective optimization to continual learning is not new, as in [1], which already explores this research direction although the application field is different.

Reference:

[1] Multi-Objective Learning to Overcome Catastrophic Forgetting in Time-series Applications. 2022

[2] A Multi-objective Multi-task Learning Framework Induced by Pareto Stationarity. ICML 2022

My major concern is w.r.t the technical part, where the current gradient update for Pareto optimality seems inaccurate.

• The solution of eq.(8) --- $g(\tilde{\lambda}^*)$, is the weighted gradient that has the maximal minimum inner product with the gradients of previous tasks, indicating a minimal worst-case interference with these tasks.
• However, $g(\tilde{\lambda}^*)$ is not related to $g_n$. Current update $u^*=g_n + g(\tilde{\lambda}^*)$ is not the solution to eq.(2) or eq.(3) without considering $g_n \cdot g(\tilde{\lambda}^*) > 0$ or $g_n\cdot g(\tilde{\lambda}^*) <0$.
• To consider a similar objective, it needs to optimize $\max_u\min_{\alpha \geq 0} -\frac{1}{2} ||g_n -u||_2^2 + \alpha \langle u, g(\tilde{\lambda}^*)\rangle$. This gives $u^*=g_n + \alpha^* g(\tilde{\lambda}^*)$, where $\alpha^* = \arg\min\_{\alpha} \frac{1}{2}||g_n + \alpha g(\tilde{\lambda}^*)||_2^2$. The optimal solution is as your discussion of RGA in Figure 3.

The paper seems to lack a thorough discussion on the practical applications and limitations of Pareto optimization in continual learning. Comparisons with other gradient alignment-based methods are primarily performance-centric, without a deep dive into the conceptual differences and nuances of each approach.

I have several major problems with the current state of the manuscript. Since the scores in my review can look quite harsh, I will try my best to explain why I opted for these scores below:

Soundness

• W1. My score is based on the several claims and comments regarding Pareto optimality and multi-objective optimization (MOO). They simply make no sense (e.g. "as a crucial manner to achieve MOO", "we introduce the Pareto optimality to..."), and some claims are plainly wrong (e.g. "[Pareto optimality] seeks to maximize the overall marginal benefit across different tasks") and make me suspect that the authors did not fully grasp the meaning of Pareto optimality (e.g. any point in the Pareto frontier is Pareto optimal, and their losses can be, in general, as (im-)balanced as desired).

• W2. On a similar note, the term "hypergradient" seems to be used quite freely here, and I haven't been able to find a reference where hypergradients are defined a way similar to how it is done in Eq. 9. The two meta-learning citations use a different gradient to the best of my understanding.

Presentation While the paper is generally well-written, there are several typos, and citations and equations are not properly formatted. However, my low score relates with the "contextualization relative to prior work" and the disconnection between the proposed methods. I will describe now my biggest concerns:

• W3. Methods are not properly cited. For example,
  • MGDA is attributed to (Sener & Koltum, 2018) instead of the original author [1]. Sener & Koltum proposed MGDA-UB, an "upper-bound" of MGDA.
  • Again, (Sener & Koltum, 2018) is referenced together with another paper to cite the Frank-Wolfe algorithm, instead of citing the paper by Frank & Wolfe [2].
• W4. It is rather unclear how the two proposed methods are related (beyond solving two GR problems), and I feel the presentation sort of "forgets" about RGA at times. Indeed, RGA completely disappears from tables 3 and 5 and their corresponding discussions.

Contributions My bigger concerns here are three important ones:

• W5. GR reformulation I fail to see reformulating the problem as gradient re-weighting as a huge contribution as, to my understanding, the original derivations from (Lopez-Paz & Ranzato, 2017, Eq. 11) already define the problem as a GR problem.
• W6. RGA (minor) While the derivations are not super complicated, I find quite funny that all the derivations from section 3.1 are an almost one-to-one with those derivations from [3, App. A.3] (that solves a slightly-different problem)
• W7. PRGA This one I find it super concerning. PRGA (understood as the derivations from section 3.2) is far from novel, as it is exactly the MGDA algorithm, popularized by Désidéri in 2012 [1], brought to the MTL community by Sener & Koltum in 2018, and first discussed (to the best of my knowledge) by Fliege & Svaiter in 2002 [4] (indeed, Eq. 5 is equivalent to Eq. 2 in their paper).

Experiments I have mostly focused on the theoretical part of this work, so I hope that my peers can complement my review in that regard. However, I have two concerns:

• W8. Given my discussion above, the effectiveness of the proposed methods seems to be a well-executed combination of existing techniques (which is valuable!). As such, it would be important to understand the degree of importance of each of these components, but I am afraid that the "iterative" ablation study from table 4 does not provide the necessary full-picture.
• W9. Again, I have not thoroughly looked into the different hyperparameters, but a quick look into papers with code shows that the results for CIFAR100 with 20 tasks in CL has quite higher average accuracy (being the worst there 67%, and the best one here 33% in Table 1). As such, it would be necessary to clarify why such a difference in results, and whether conclusions hold despite the substantially lower performance.

[1]: Désidéri, J. A. (2012). Multiple-gradient descent algorithm (MGDA) for multiobjective optimization. Comptes Rendus Mathematique, 350(5-6), 313-318.

[2]: Frank, M. and Wolfe, P. An algorithm for quadratic programming. Naval research logistics quarterly, 3(1-2): 95–110, 1956.

[3] Liu, B., Liu, X., Jin, X., Stone, P., & Liu, Q. (2021). Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34, 18878-18890.

[4] Fliege, J., & Svaiter, B. F. (2000). Steepest descent methods for multicriteria optimization. Mathematical methods of operations research, 51, 479-494.