Freeze and Cluster: A simple baseline for Rehearsal-Free Continual Category Discovery

1. Some conclusions are not convincing. For example, the paper contends that We believe that continuous learning with unlabeled data accumulates noise, which is detrimental to representation quality. The results might come from the limited exploration of combination methods. In rehearsal-free continual learning, feature-replay methods have shown great potential, like [R1] in continual learning and [R2] (FRoST) in CCD. A more recent work [R3] also employs feature replay to continually adjust the feature space, which also obtains remarkable performance for continual category discovery.
2. The proposed method is naïve and the novelty is relatively limited. The method includes basic clustering and number estimation. I’m afraid that this method could not provide so many insights to the community.
3. The feature space (i.e., backbone) is only tuned using labeled known classes, does this manner result in overfitting? because the data is purely labeled but the number is limited.
4. The class number estimation algorithm requires a pre-defined threshold $d_{min}$, which is intractable to define in advance and could largely impact the results. Some experiments and ablations should be included.
5. Detailed results of each continual session (at least for one or two datasets) should also be presented to show the performance.

References: [R1]. Prototype Augmentation and Self-Supervision for Incremental Learning. CVPR 2021. [R2]. Class-incremental Novel Class Discovery. ECCV 2022. [R3]. Happy: A Debiased Learning Framework for Continual Generalized Category Discovery. NeurIPS 2024. arXiv:2410.0653.

1. The interpretation of the observations in Fig. 1, as presented in L287-295, may not be entirely accurate. The primary objective of this experiment is to evaluate whether the unlabeled data can enhance the pre-trained representation. When trained with a supervised loss, the backbone aligns with the testing protocol by utilizing true labels, naturally converging to a state that reflects the task’s labels. In contrast, the unsupervised SeDist loss introduces discrepancies, as it neither aligns with the pre-training objective nor the evaluation classes, disrupting the pre-trained knowledge. Consequently, the learned representation is likely degraded due to the lack of constraints. To draw more concrete conclusions, it would be advisable to replace SeDist with the original pre-training loss function to maintain consistency between the two stages [A]. Additionally, it would be beneficial to conduct experiments in an incremental setting where both old and new classes are incorporated.
2. The novelty of this paper is somewhat limited. While I acknowledge that the authors achieve better results compared to previous methods, as shown in Table 2, the proposed approach is relatively simple and intuitive. The main motivation of the paper centers on the observation that earlier methods do not leverage pre-trained models, and updating the backbone during incremental sessions degrades pre-learned knowledge. The solution proposed, utilizing a pre-trained model and freezing the backbone, is straightforward. In the field of continual learning, the challenge of balancing plasticity and stability has been extensively studied. Increasing the number of trainable parameters enhances plasticity but compromises stability, while reducing the number of trainable parameters has the opposite effect. This trade-off is well understood and has been widely explored in the literature. Freezing the backbone is a reasonable strategy to preserve pre-trained knowledge, but it limits the model's capacity to learn new information during incremental sessions. The authors are encouraged to explore more sophisticated balancing strategies, such as those used in methods like Grow and Merge, to better address the plasticity-stability trade-off.

Writing needs improvement:

1. In L21-24 of the abstract, the claim regarding performance improvement through better integration of previous methods is not directly relevant to the conclusion that unlabeled data can harm the learned representation.
2. In L58, the authors critique prior works for not addressing the unique challenges of RF-CCD. However, the paper does not explicitly define what these unique challenges are. It would be beneficial to reorganize the discussion, particularly with paragraph L60-71, to provide a clearer explanation.
3. Additionally, the introduction does not sufficiently explain how unlabeled data can degrade the representation. It would be helpful to explicitly mention that updating the model using unlabeled data can disrupt the pre-trained backbone, as this critical detail is currently missing.
4. It is recommended to establish a clearer connection between the conclusion in L300 and the methodology described in Section 4. For example, the writing should explain how the conclusion informs or motivates the proposed method.

[A] Finetune like you pretrain: Improved finetuning of zero-shot vision models. CVPR’23

1.It seems that the proposed method is a simple combination of existing continual learning and novel class discovery methods for RFCCD, which is somewhat not novel.

2.More related GCD works should be included for comparisons, i.e., [1-3]. And I wonder what performance can be achieved when combining the best GCD and continual learning method for this task.

3.The datasets used for evaluation are somewhat small. The results on more large-scale datasets are more convincing (i.e., ImageNet-1K).

[1]Online Continuous Generalized Category Discovery

[2]PromptCCD: Learning Gaussian Mixture Prompt Pool for Continual Category Discovery

[3]Happy: A Debiased Learning Framework for Continual Generalized Category Discovery

There are some minor issues that will not affect the rating: 1."continuously identifying novel class by " should be "continuously identifying novel classes by ".

2."Specifically, for known classes data" should be "Specifically, for known-class data".

There are several technical concerns as listed below in "Questions".

A discussion on the computational efficiency of "Freeze and Cluster" is needed.