Multiple Modes for Continual Learning

1. The motivation for the model design is somehow intuitive and based on toy empirical results.

2. An important related work is missing. CoSCL [1] is a recent ensemble-based method that also train multiple parameter modes in parallel and regularize their diversity, implemented with regularization-based baselines. I find many empirical analyses between this paper and CoSCL are similar, such as the trade-off between learner number and size (Figure 2), and the flatness of loss landscape (Figure 3). To demonstrate the contributions of this paper, an in-depth comparison is strongly encouraged.

3. Although the proposed method can improve EWC to some extend, the performance lead against the strongest baseline (i.e., WSN) seems to be limited and is demonstrated in relatively simple benchmarks (i.e., Task-IL and CIFAR-100). Experiments in more complex benchmarks will be more supportive.

4. All experiments use a pre-trained checkpoint on ImageNet that overlaps with downstream datasets such as CIFAR-100 and Tiny-ImageNet.

[1] CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One. ECCV 2022.

I believe that the primary issue with this work is its lack of clarity and organization. It is quite challenging to read, and I attribute this difficulty not to any inherent technical complexity in the proposed method but rather to its poor presentation. I find this to be particularly unfortunate and frustrating because, in my opinion, the work has the potential to make a valuable contribution. It's worth noting that the manuscript falls almost a page short of the total length allowed for this venue, and utilizing that additional space could have helped address some of the main concerns I will now list (in no particular order).

1. In many parts of the manuscript, you refer to 'global geometry.' Sometimes, you speak of the global geometry of the parameter space, while at other times of the loss landscape. Although these concepts may be related, they are not the same thing. Furthermore, there are instances where you use the term 'global geometry' generically. It would greatly enhance clarity if you provided a formal definition of what you mean by 'global geometry.'

2. A complexity analysis is necessary to understand how the method scales. The method's update process involves a backtracking algorithm, and some of the loss calculations require computing distances between sets of weights from different models for each task. A discussion of the computational complexity would provide valuable insights.

3. I really do not understand either the statement nor the implications of Theorem~1; in the statement there are either undefined symbols like $T(i)$ or odd notation like $i\in N$, when $N$ it has been previously define as a natural number and it is clearly not a set (similarly for $T$).

4. No explanation/justification has been given to why the $\alpha_i$ are randomly sampled nor the specifics of their random generation. Why it is better than posing $\alpha_i=1/N$ for all $I=1,\dots,N$, how does this sampling affects performances?

5. The authors choose to update the parameters of the various models to minimize the drift from previous parameter values. While I partially understand this approach, it seems to be a critical point of criticism the authors had against EWC. A more comprehensive discussion of this point is needed.

6. The paper's organization is suboptimal. Section 2 combines background information with motivations and results, while Section 3 presents crucial aspects of the proposed method more as documentation for the 'initialize_parameters' and 'update_parameters' methods than as an organic organization of concepts.

7. The manuscript lacks a discussion of the signs of $\beta_{\rm max}$ and $\beta_{\rm min}$, even though it is crucial. Based on the pseudocode, if $\beta_{\rm max}$ is positive, it effectively minimizes the distance, while the text suggests that it maximizes it. A clarification regarding the sign and its implications is needed.

1. The initial task seems to be important as the modes are computed initially using the first task. Experiments about this should be included.
2. The authors should report the training time of the proposed method as well as that of all the baselines. Seems like the proposed method is computationally expensive as it computes the gradient of the parameters in each mode in a task (line 7 Alg.1), and it computes another gradient for the combined parameter (line 12 Alg.1).
3. The task split is not clear. The authors said ``in instance-IL, each new task brings new instances from known classes”, but Instance-IL Split-CIFAR100 is constructed by dividing 100 classes into 5 tasks with 20 coarse labels per task. From my understanding, each task is disjoint from each other.
4. I personally think the main CL setups are class-incremental, task-incremental, and domain-incremental learning. If the proposed method targets a different learning scenario, the authors should have made the definition of the scenario more clearly. From the current manuscript, it's not clear how Instance-IL is constructed.
5. The authors said the method is online. However, according to the Experiment Configuration of the Appendix, the model is trained for 200 epochs. Is this online CL, where the model is trained for a single epoch per task or is it offline, where the model is trained for multiple epochs?
6. It's concerning that the authors use a network pre-trained with the full-ImageNet data to train for learning CIFAR and Tiny-ImageNet. A lot of the classes in ImageNet are very similar to the CL datasets CIFAR100 and Tiny-ImageNet. Therefore, there could be information leaks from the pre-training classes to the CL classes. For this reason, many existing methods ensure that the pre-training and CL classes are different. For instance, [1] uses a pre-trained model pre-trained with samples in ImageNet after removing the classes similar or identical to the CL datasets. [2, 3] pre-train their model for half of the classes (e.g., 50 for CIFAR100) and continually train the model with the remaining classes.
7. The experiment results are not surprising. The task-IL experiment result seems a bit low. [4, 5] achieved more than 80% accuracy and [6] achieved more than 92% accuracy.

[1] Learnability and algorithm for continual learning
[2] Prototype Augmentation and Self-Supervision for Incremental Learning
[3] Dynamically Expandable Representation for Class Incremental Learning
[4] Overcoming Catastrophic Forgetting with Hard Attention to the Task
[5] Supermasks in superpositions
[6] A theoretical study on solving continual learning

1. During the experiment, the author majorly compared with EWC, I think some recently work that focused on similar idea (not only parameter drift), e.g., ensemble on network. Should also be discussed and compared, for example, Continual Learning Beyond a Single Model, Dynamic Network Expansion and so on [1,2].

2. Some unclear description, e.g., a task has a high level of certainty/uncertainty, how could we measure the degree of such tasks? On sec 3.1, for updating each mode with respect to the input loss, what's the input loss here? The cross-entropy?

3. For the experiment, it do not cover the commonly used Class-IL, or domain-IL setting. Also, for the difference and discussion between task-IL and instance-IL, the author did not describe them more clearly.

[1]. Efficient Continual Learning Ensembles in Neural Network Subspaces. [2]. BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning. ICLR 2020.