Learning Forward Compatible Representation in Class Incremental Learning by Increasing Effective Rank

1. Overall, I find the intuition to utilize effective rank to enhance forward compatibility is not well illustrated. There are many core factors that are not well discussed in the main paper, e.g., the relationship between feature richness and effective rank, the relationship between feature richness and forward compatibility, and the relationship between feature richness/effective rank and catastrophic forgetting. Hence, reading the paper sometimes makes me confused about why the heuristic loss works for the specific capability.
2. As repeated many times in the main paper, the authors argue that CwD [1] and FACT [2] require “class information” to enhance forward compatibility. What does the “class information” mean in this paper, since the concept is abstract and hard to figure out? In my opinion, CwD and FACT can work during the first incremental stage and use only the base classes (first-stage classes) to enhance forward compatibility. They do not require extra information like future classes to design the loss term, which is almost the same as the authors do in Eq. 5. Hence, it requires careful clarification about this claim point.
3. As another noteworthy weakness, I must highlight that the current experimental evaluation is somehow weak, lacking the comparison on ImageNet1K (dataset) to other state-of-the-art works like DER, FOSTER [3-4] (methods), other related competitors like FACT, BEEF [5] (methods), and the experimental results without half base classes (settings). See more comments in the following questions.

1. The RFC loss is only used during the training of the base task and does not exhibit significant improvements when applied also to novel tasks. While the authors acknowledge that using this regularization intensifies model modifications from the previous session, a more detailed analysis of this limitation is essential since it represent a crucial problem in class incremental learning.

2. Many state-of-the-art methods cited in the paper that benefit from the RFC loss are backward-compatible exemplar-based approaches. However, in recent times, exemplar-free methods have emerged as strong competitors. They rely on prototypes instead of examplars and often employ Class Augmentation techniques during the base task to enhance the initial feature representation. For example, [1] employs a mixup approach to achieve a similar effect to the proposed RFC loss. In fact, in that paper, the authors show that their method can increase the number of directions with significant eigenvalues in the feature space. Other works, such as [2], follow similar reasoning by using self-rotation as class augmentation technique. In my opinion, a comparison with these exemplar-free methods could be valuable to understand whether RFC could be used in substitution or synergy with these Class Augmentation techniques.

3. The paper could benefit from the inclusion of a dedicated section addressing the limitations of the proposed approach. This section would provide transparency and a more thorough assessment of potential shortcomings, allowing readers to understand the practical constraints and challenges associated with the RFC loss method.

[1]: Class-Incremental Learning via Dual Augmentation, NIPS 2021

[2]: Prototype augmentation and self-supervision for incremental learning, CVPR 2021

• The novelty of the paper is limited. Although the proposed specific method may be different from others, the idea is not new, and many existing methods have been proposed to achieve the same effect. Existing methods mainly achieve it by using contrastive learning and/or data augmentation, which are not compared in this paper but should be. The following paper specifically tries to increase the richness of the learned features,

[1]. Class-Incremental Learning via Dual Augmentation. NeurIPS, 2021.

• The following paper considers both forward and backward as it uses out-of-distribution detection. It also uses contrastive learning.

[2] A theoretical study on solving continual learning. NeurIPS, 2022.

• The base task in the paper is a form of pretraining. Existing method using a pre-trained model should be compared, e.g.,

[3] Lifelong machine learning with deep streaming linear discriminant analysis. CVPR workshops, 2020. [4] Learning to prompt for continual learning. CVPR, 2022 [5] Learnability and algorithm for continual learning. ICML 2023.

• The accuracy results based on average incremental accuracy is very low, may be lower than accuracy after learning the last task of some SoTa methods.

• The baselines used in the paper are old.

• More datasets should be used in the experiments.

• Compared methods. The used baselines in this paper is old. iCaRL, EEIL, BiC, UCIR are early-stage exemplar-replay methods, and stronger baselines like DER are suggested to included. Similarly, LwF, MAS, SI, Rwalk are early-stage regularization methods, and stronger non-exemplar based methods are suggested to be included.
• Experiments. How the proposed method performs on large scale dataset ImageNet Full?
• Novelty. The proposed method is the same as vne proposed in (Kim et al., 2023), which may lack of novelty. In additional, the idea of increasing rank for incremental learning has been explored in the paper of CwD (Shi et al., 2022).