Out-of-Distribution Detection in Class Incremental Learning

1. The out-of-distribution dataset in OpenCIL benchmark is static, that is, these samples or classes remain as out-of-distribution throughout the whole incremental learning process. It does not address the scenario in the open-world where the incremental learner learns the out-of-distribution samples detected. In this case, these samples should no longer be detected as out-of-distribution by the model in the next incremental phase.
2. The main findings stated in Section 3.2 needs further clarification. There is no strong justification of why 'catastrophic forgetting is more persistent in CIL models than OOD detection methods' given that the authors are comparing the decreasing rate of two different metrics. It is also unclear how the authors justify that old class samples are detected as out-of-distribution samples. From the prediction confidence in Figure 1, it is difficult to understand how the authors claim that old class samples have lower prediction confidence than out-of-distribution samples.
3. There seems to be some conflict proposed regularization terms. In Eq (3), the model penalizes in-distribution samples that produce energy lower than $p_{in}$ while penalizing pseudo-OOD samples that produce energy higher than $p_{out}$. On the other hand, in Eq (5), the model is penalizing augmented in-distribution samples that produce energy higher than $p_{in}$.
4. Limited class-incremental learning methods are used as baselines, where the most recent method is from year 2022. For a more comprehensive evaluation, it would be beneficial to include some of the latest released methods to better demonstrate the robustness and generalizability of proposed method.
5. Novelty is limited as the proposed method is a simple combination of mixup augmentation with the energy-based out-of-distribution detection from previous work (Liu et al., 2020).

1.Lack of Comprehensive References: The paper does not thoroughly cite recent studies in the field, particularly the latest methods in Class Incremental Learning (CIL). The comparison experiments use CIL methods with the latest from 2022, which may not reflect the most current advancements. More comprehensive references and comparisons could enhance the credibility and persuasiveness of the approach. 2.Unclear Equations: In Sections 4.1 and 4.2, the key parameters pin and pout are not explained, which may hinder readers’ understanding of the equations. Additionally, there is a labeling error in Algorithm 3 in the Appendix, where Eq. 2 is mistakenly labeled as Eq. 1. Correcting these will improve the paper's professionalism and readability. 3.Lack of Advanced Comparison Methods: The comparison experiments do not include the most recent CIL methods, which may affect the reliability of the results. It is recommended to incorporate more recent CIL methods in the comparisons to provide a more thorough evaluation of the proposed framework.

• The paper briefly mentions hyperparameters like $\tau$ and $\alpha$ for the BER approach, but there’s no thorough exploration of how sensitive BER is to these parameters. A parameter sensitivity analysis could offer insights into how to optimize BER for different datasets and models, making it more user-friendly.
• the complexity of the approach might pose challenges for real-world implementation, especially regarding computational overhead due to the additional synthesis and regularization components in NTER and OTER.
• There are missing citations in the manuscript. For example, the paper introduces generative models, but generative-based methods [1, 2, 3] are missed without corresponding details in the bibliography.

[1]: Kong, Shu, and Deva Ramanan. "Opengan: Open-set recognition via open data generation." In ICCV. 2021.

[2]: Wang, Qizhou, et al. "Out-of-distribution detection with implicit outlier transformation." In ICLR, 2023.

[3]: Zheng, Haotian, et al. "Out-of-distribution Detection Learning with Unreliable Out-of-distribution Sources." In NeurIPS, 2023.

1. The motivation for benchmarking OOD in CIL is not strong, as OOD detection becomes more and more ineffective during CIL is intuitively predictable. There is no need to spend a lot of resources to illustrate a predictable finding.
2. Although the mixup-based pseudo-OOD samples and synthetic old task data are interesting, the OOD detection of the proposed BER still relies on existing energy-based methods. In terms of technical novelty, it is more appealing to propose a new OOD scoring mechanism dedicated to CIL.
3. For the OOD classifier, how to initialize it is unclear. Is it initialized from the ID classification classifier or inherited from the latest old OOD classifier?
4. I would disagree with the finding of lines 337-339. In most cases, the performance of fine-tuning-based OOD and post-hoc-based OOD methods are comparable. Besides, I am not sure why BER is only compared with fine-tuning-based OOD approaches. There are some cases in which BER cannot beat all post-hoc-based OOD methods. If fine-tuning-based BER cannot far exceed post-hoc-based OOD methods, I cannot figure out a reason to adopt BER, which even consumes much higher computation costs.
5. The paper presentation and organization have a lot to improve. For example, in Figure 3, there is no legend for dashed curves in subfigure (a); the legend in subfigure (b) denotes ACC performance and different OOD methods, which is confusing. Besides, the curves are the same type; The same curve with the same color denotes different methods among the three subfigures, which is hard to distinguish and read. In addition, the last finding (starts at line 337 on page 7) refers to Tables 1,2,3 that are on pages 9 and 10, which largely hurts the readability.