Is multitask learning all you need in continual learning?

We thank the reviewer for the opportunity to clarify the core idea of the paper and its philosophical standpoint. Prompted by this review, we have revised the introduction of the paper, insisting on the motivation behind our specific choices. We believe that these changes have greatly improved the clarity of the paper.

The reviewer correctly points out that "in continual learning, we want to train a model first on one task, then on a second task, and expect the model to perform well on both tasks simultaneously." This is what we mean when we say that continual learning algorithms implicitly aim at optimizing a multi-task objective ('performing well on all tasks' is another way to describe the MT objective). However, the question that our paper is posing to the continual learning community is whether this multi-task objective in practice achieves the end goals of the community. A good follow-up question is "what are the goals of the continual learning community?" In the paper, we follow recent literature ([1], [2], [3]) in recognizing that the ultimate goal of a continual learning agent is to perform well throughout its lifetime. When a system is deployed in the real world, it should leverage its previous knowledge to enhance its current performance, e.g., by learning faster, performing better, and retaining useful knowledge as long as it is beneficial. In this sense, remembering serves the purpose of enhancing current and future performance. Kumar et al. ([2]) have formalized this statement using information theory, showing theoretically that forgetting of 'useless' knowledge is not a bad thing when one cares about lifelong performance.

Therefore, it is true that "Using the 'average lifelong error' metric places multi-task agents at a disadvantage" but -as you correctly notice- this depends on the correlation between the tasks, a concept which we formalize with our definition of "instability". This forms the main claim of our paper: that multi-task learning is not always optimal for lifelong performance. However, single-task learning (i.e., always erasing memory) is not always optimal either.

Perhaps the most exciting consequence of this statement is that the design of optimal CL agents (where optimality is defined based on lifelong performance) requires objectives which are data dependent. In other words, if we want to build better CL agents we should develop methods to estimate the optimal amount of forgetting for a given sequence of tasks, which is a clear direction for future research.

We acknowledge that the main claim may appear simple and overemphasized. However, we believe it has important implications for the field, and developing the experimental and theoretical framework to support the claim was not trivial.

Overall, we understand that the reviewer may disagree with us on a philosophical level, as to what is supposed to be the main goal of continual learning. And rightfully so: this is an open question and a topic of ongoing debate within the CL community. However, this paper is not trying to argue about the "true" goal of continual learning, but rather, it takes a specific stand in this philosophical debate as a starting point. We hope the reviewer can appreciate the solidity of our work beyond the philosophical dispute.

If we have answered the reviewer's question we kindly invite the reviewer to briefly check the introduction in the new version and provide feedback regarding the clarity of our motivation and positioning within the continual learning community. This aspect of our work is really essential to understand its value and so we would like to make sure that the message is conveyed correctly.

[1] Hadsell, Raia, et al. "Embracing change: Continual learning in deep neural networks." Trends in cognitive sciences 24.12 (2020): 1028-1040.

[2] Kumar, Saurabh, et al. "Continual learning as computationally constrained reinforcement learning." arXiv preprint arXiv:2307.04345 (2023).

[3] Mundt, Martin, et al. "A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning." Neural Networks 160 (2023): 306-336.

We thank the reviewer for their thoughtful response and active engagement in the rebuttal process. The points raised are certainly valid, and we find this discussion both stimulating and highly relevant to the community. We understand your reservations, particularly regarding the impact of our claims and the framing within the continual learning domain. Below, we would like to address your concerns in more detail:

As noted by the reviewer, we have revised the manuscript to emphasize our focus on forward transfer rather than backward transfer, as this is central to our study. This focus naturally brings us closer to online learning, which traditionally emphasizes forward transfer. The core of this discussion is whether our work is more aligned with continual learning or better categorized as online learning. We believe this is largely a philosophical question that ultimately leads to differing interpretations of the goals of continual learning. There are two main perspectives in this debate: a traditionally dominant view, which defines continual learning as the challenge of balancing forward and backward transfer, and a more recent perspective, which sees continual learning as the ability to perform well throughout the lifetime of the agent. In this work, we adopt the latter view, which aligns continual learning with fields like online learning and reinforcement learning. For example, Kumar et al [1]. frame continual learning as "computationally constrained reinforcement learning". We view this alignment with other fields positively, as the exchange of ideas and tools across disciplines can accelerate progress. One implicit goal of our work is indeed to bridge the gap between online learning and continual learning. Nonetheless, key differences remain: historically, online learning has been more theoretical, often relying on strong assumptions such as convexity, whereas continual learning has focused more on deep learning with an emphasis on practical applications.

The reviewer mentioned that focus on forward transfer alone makes our findings less impactful, given that the benefits or detriments of out-of-distribution data are well-understood in the community. We recognize the validity of this observation. However, our intention is to clarify specific scenarios in which multitask objectives may lead to suboptimal forward transfer performance. For example, the concept of instability introduced with our theoretical analysis, allows for a quantitative assessment of the suboptimality of multitask objectives in a given environment. We believe that highlighting these nuances, while not groundbreaking, offers a concrete basis for the development of provably optimal objectives for lifelong performance.

Additionally, as summarized in the Discussion and Conclusion section, our argument is that "reliance on the multitask objective is not necessary" and that the optimal approach should depend on the data. This is not equivalent to "abandoning multitask approaches altogether"; rather, a multitask objective should be employed when it is optimal and set aside when it is not. More broadly, we envision that future research in continual learning should focus on defining data-dependent objectives that dynamically determine the appropriate balance of memory for the agent. Finally, given the philosophical nature of this discussion, unless the reviewer finds technical flaws in our work, we do not see a substantial basis for recommending rejection.

[1] Kumar, S., Marklund, H., Rao, A., Zhu, Y., Jeon, H. J., Liu, Y., & Van Roy, B. (2021). Continual Learning as Computationally Constrained Reinforcement Learning. Advances in Neural Information Processing Systems (NeurIPS).

Thank you for your detailed and insightful comments. We appreciate the opportunity to address your concerns and provide further clarifications.

1. We have improved our toy settings significantly based on your feedback. The revised toy setting now provides a clearer and more nuanced demonstration of the key factors identified by our theoretical analysis. Specifically, we refined the mathematical formulation in the appendix (Section A.3) to better reflect more realistic scenarios where a multitask loss solution is applicable, addressing your valid concern about contradictory signals. This setting demonstrates that, even when a multitask loss is viable, it does not guarantee optimal performance.

2. We have clarified the structure of the empirical evaluation section. As per your suggestion, we added a description at the beginning of the section to explain the motivation behind splitting the experiments into two parts: the CLEAR dataset and MD5 vs. CIFAR10 Permuted. Specifically: The CLEAR dataset, MD5 and ML10 experiments focus on validating our main claim (that the MT objective is not always better) in realistic benchmarks which are used in practice.

   The CIFAR10 Permuted experiments are meant to validate more specific and nuanced predictions from our theory, which can only be performed in environments where we can control the instability $\Delta_T^I$. Since permutations of the input significantly increase the learning task difficulty, we opted for a simpler task such as CIFAR10.

   This restructuring helps clarify the rationale for including both parts and the insights they provide.

3. Regarding the connection to the online CL literature, we acknowledge that there was an important part of related work missing from the discussion. Prompted by your observation, we have added a paragraph (in Section 2) regarding the OCL literature and the connection between existing metrics and our average lifelong error. In particular, the so called 'average anytime accuracy' (as defined by [1]) should be equivalent to $1 -$ our average lifelong learning error, and it also corresponds to the online average accuracy (defined by [2]). Additionally, we reiterate that the average lifelong error is directly linked to another crucial metric in online learning -the dynamic regret.

   In order to make our empirical results more easily accessible to the rest of the community, we have modified the experimental sections and we now report accuracies instead of errors.

4. We have clarified and softened the language around the claim that "most CL methods use the ERM objective." We acknowledge that the claim has been taken almost for granted in the paper, while it is not common knowledge and requires justification. We have now revised our discussion in the main paper and added an in-depth review of existing CL algorithms and their link to the MT objective in the appendix (Section B in the new version). In general, CL algorithms do not implement the exact MT loss. Instead, they can often be viewed as biased estimates of it, where the MT loss is either weighted differently across tasks or approximated by regularization terms that may deviate from the original objective. We still believe that studying the MT objective offers a high level intuition on many continual learning algorithms. However, we acknowledge that the precise level of 'suboptimality' for specific CL methods is not clear. To determine this, our theoretical framework has to be applied to the specific objective of each method.

5. Regarding the description of the permutation experiments, it can be found in the appendix (section C.1.4., Figures 8 and 9 in the new version).

[1] Soutif–Cormerais, Albin, Antonio Carta, and Joost Van de Weijer. "Improving online continual learning performance and stability with temporal ensembles." Conference on Lifelong Learning Agents. PMLR, 2023.

[2] Cai, Zhipeng, Ozan Sener, and Vladlen Koltun. "Online continual learning with natural distribution shifts: An empirical study with visual data." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

We thank the reviewer for acknowledging our efforts and for the stimulating discussion. We would like to continue the conversation regarding some of the ideas raised. The reviewer suggests that, to shift the community’s focus toward adaptivity, we should redefine existing metrics such as ‘forgetting’ to align with this focus, for instance, by redefining forgetting in terms of the speed of adaptation when an old task is encountered. While we understand the reviewer’s point, we believe that reinterpreting existing metrics could lead to confusion. That said, we fundamentally agree with the reviewer that, in the context of forward transfer, lower forgetting of past tasks correlates with faster adaptation when a task is revisited. Intuitively, minimizing forgetting becomes particularly important when the likelihood of encountering a task again is high. Thus, incorporating assumptions about the probabilistic structure of the environment could help determine the appropriate level of forgetting. This aligns with our proposition that future research in continual learning (CL) should focus on developing data-dependent objectives that are provably optimal.

The goal of this work is to demonstrate that the currently prevalent multitask objective is suboptimal, thereby opening the door for discussions like the one we are having in this rebuttal and encouraging future research on alternative objectives for CL. Finally, we would like to note that our theoretical analysis can account for ‘task revisitation’ as it does not make assumptions about task similarity. Specifically, task revisitation would reduce the instability of the task sequence, depending on the ordering. In the toy setting presented in the paper, the same two tasks are revisited alternately in a repetitive manner.

We would like to thank the reviewer for the thoughtful and constructive feedback on our paper. We appreciate the opportunity to clarify our work and make improvements based on your comments. We would like to point out that we have switched the order of the sections in the Appendix, the theoretical results now come first.

1. Regarding the main caveat of the work, we believe there is a misunderstanding. We have modified Section 3 and 4 in order to improve the exposition of our theoretical analysis, and we hope to have resolved this confusion.

   In particular, the MT agent (which can be seen as a 'perfect' replay agent with infinite buffer, representing an agent that minimizes forgetting) is trained on the "average error across all tasks encountered up to the current point" (line 155 in the revised paper), and indeed in the appendix (now A.1.2.) the MT objective during task $\kappa$ is equal to the sum of errors $R_i$ where $i\le \kappa$. Thus, the MT GD agent 'continually descends', meaning that it is not reset at the end of a task, but keeps its previous task estimate as initialization. The assumption which you mentioned in line 196 (line 210 of the new version), specifically $\frac{1}{K} \Delta_T^{MT} \in o(1)$, is proven in the convex setting in Lemma 9 (of the new version). It is a simple consequence of the fact that the contribution of each task to the new average decreases as the task index increases. We also recognize that the previous definition of key quantities in the theoretical sections was sometimes sloppy, so we have thoroughly polished Section 3 and 4. We kindly ask the reviewer to have a look at the new version and share their thoughts.

2. We significantly strengthened our theoretical analysis, by adding new results and claims. Specifically, we derived a matching lower bound for $\Delta T$, which allowed us to establish tighter bounds, and explicitly describe the dependency on h. Additionally, we expanded our derivations to cover the overparameterized case, and refining the previously inconsistent exposition of theoretical results. We believe these improvements consolidate the theoretical foundation of our work.

3. The results in Table 3 are intended to evaluate the effect of increasing the number of tasks. From our theoretical result the variable $K$ only indirectly affects the balance between MT and ST objectives, through the instability. Thus, the experiments of Table 3 are meant to evaluate in practice what the effect of $K$ is. We have revised the presentation of Table 3, and, more generally, Section 5 to make the underlying hypotheses more explicit and easier to follow.

4. We have added a detailed description of the Selective Replay algorithm in the main text to address the concerns about its omission. This addition should provide better context for understanding the experiments and their outcomes. We thank the reviewer for highlighting this point.

We thank the reviewer for their valuable suggestions, which have significantly helped improve the paper. Below, we address the specific points raised:

1. We have improved the overall presentation, especially in Sections 1 to 4. We also added the definition of $\theta^\star$ at the beginning of Section 4. We have corrected various typographical errors, including inconsistencies in notation, grammatical issues, and minor formatting mistakes that were present throughout the manuscript.

2. You correctly noted that our theoretical results are derived under the assumption of linear models in a strictly convex setting. This limitation constrains the generalizability to overparameterized models. We have extended our theoretical analysis to discuss how to handle the overparametrized case in greater detail, which can be found in Section A.4 in the Appendix material. Importantly, the results do not change at a fundamental level.

3. The question of deciding between objectives is indeed fundamental to continual learning, and our paper emphasizes this. We acknowledge that the current estimates of instability may be impractical due to their computational cost; however, in many cases, a task transfer matrix or equivalent task structure information could suffice. We believe this presents a promising direction for future research. To explore this further we attempted to measure instability at initialization using the Neural Tangent Kernel (NTK). The results are preliminary, and thus we opted not to include them in the main text, but they can be found in Section A.5. We look forward to your feedback on this aspect.

4. Regarding the confusion about the online versus offline learning nature of the training: our setup is indeed atypical, and this is intentional. Our goal is to determine whether the multi-task objective is optimal with respect to a lifelong performance metric, such as the average lifelong error (or dynamic regret). While our evaluation metric is online -meaning that it only considers the current task error- our training paradigm does not need to be. We decouple the training and evaluation protocols, allowing for potentially offline training. We have clarified this distinction in the introduction and the background sections (see lines 120-130).

Dear all, as the rebuttal period nears its conclusion, we kindly remind reviewers to actively participate in the rebuttal discussion. We have implemented substantial revisions to the paper in response to the insightful feedback provided, which we believe has significantly enhanced its quality. We would greatly appreciate it if you could take a moment to review the updated version and share your thoughts by providing a response at your earliest convenience. Thank you for your valuable contributions to this process!