Response to reviewer si3G, part 2

• Q5: Clarification on why the final block of the backbone is updated.

Our model $H_{\theta}: \{\phi, f_{\theta}\}$ i.e., the self-supervised vision foundation model (in this case DINO) consists of $\phi$, an MLP projection head, and $f_{\theta}=\{f_e, f_b\}$ a transformer-based feature backbone that includes an input embedding layer $f_e$ and self-attention blocks $f_b$. During training, we only optimize $\phi$ and the final block of $f_b$. After training, we only require $f_{\theta}$ to extract the token classification i.e., $z`[CLS]`$ feature is used for clustering. If we fully freeze the backbone $f_b$, then unlike the L2P supervised continual learning implementation which has a final learnable fully connected layer for classification, our model will eventually not learn anything and act as a frozen DINO. Therefore, we further optimize the final block of the backbone to prevent this issue. For reference, we provide comparison results between frozen DINO, frozen DINO with learned L2P prompt pool and our proposed model in Appendix H. We can see that Our fine-tuned model outperforms the frozen models in all ACC metrics and for all datasets. Moreover, our fine-tuned model also appears to generalize better to data distributions (C100 & CUB200 datasets) that the DINO foundation model has not encountered before, which further justifies the design choice of our method for CCD.

• Q6: Clarifications of our GMP module.

1. We thank the reviewer for their careful review, the number of components in GMM is similar to the number of categories i.e, $C$ and the top-K prompts are selected among these $C$ Gaussian components.
2. To optimize GMM, we make use of classification token $z`[CLS]`$ extracted in the “gradient detach” mode of our own backbone $f_{\theta}$ as GMM samples.
3. Based on the ablation study the optimal number of samples is 100 samples per component.
4. For clarity we further provide a pseudo code of PromptCCD w/GMP model training in Appendix A.

• Q7: Clarification on query functions used on our proposed model.

As explained in R-si3G-Q6 Clarifications of our GMP module sub-section (2) we make use of our own backbone model $f_{\theta}$ as the query function to extract the classification token $z`[CLS]`$.

Response to reviewer si3G, part 1

We would like to thank reviewer si3G for their careful review, insightful feedback, and acknowledgement of our novel and intuitively sound contributions, and extensive experiments. Here we would like to clarify some concerns addressed by reviewer si3G:

• Q1: How GMM’s mean components guide CCD model.

The motivation of our proposed model is to design a prompt module that can guide or instruct a self-supervised vision foundation model for better CCD. Moreover, not only can it guide the model, but we also believe there is a need to design a prompt module that suits the open-world nature of CCD with unlabelled data that contains unseen categories over time. Thus, intuitively we choose the Gaussian mixture model (GMM) as the candidate for our prompt design. By using the GMM mean component, it acts as a class prototype (to accommodate the absence of label information) which guides/enforces the model about the criteria / to look at (information/features that the model should care about) when making the decision to discover a category. We further show in Appendix I, that our GMM coupled with self-supervised vision foundation models actually produce prompts that are discriminative within different classes. Moreover, we conduct an experiment in Table 14, where instead of taking the “top-k” mean components/prompts, we vary the prompts “random-k” to observe the effect of difference prompt relevancy towards overall CCD performance. We can see that varying the prompt hurts the model’s performance, especially for the "NEW" ACC i.e., novel categories.

• Q2: Limitation of L2P and DualPrompt for CCD task.

Thanks for the comments. Please refer to General response - Clarification on L2P & DualPrompt vs Our GMP.

• Q3: Clarification on how GMM samples are used to prevent catastrophic forgetting.

Indeed, our method can be viewed as a replay method, unlike existing methods, it does not need to store any samples, but the parameter-efficient GMM can be used to generate an infinite number of replay samples on the fly. Also, we use our replay differently than the usual replay-based method in the continual learning literature. Here the replay data is generated randomly by our GMM in each stage for each component and will be used to fit the next GMM which eventually builds/transfers previous GMM components which eventually leads to the formation of component means a.k.a prompts into the current stage GMM. Thus, by prompting these “previous components” from the previous replay samples, our model can tackle catastrophic forgetting.

• Q4: Rationale explanation regarding Table 3, main paper results.

Thank you very much for this comment. We followed this suggestion to investigate the cause of this phenomenon. While investigating the phenomenon between the TinyImageNet results obtained from both Table 2 and Table 3, we discovered a technical flaw in Table 3’s experiments Specifically, an incorrect training configuration was mistakenly loaded for training when using the GCD’s category number estimator. We have loaded the correct training configuration to retrain all models and updated the results in Table 3 accordingly. Based on the comparison of our model with other models in the unknown K settings in Table 3, we can see that our model outperforms all other models on all datasets under all metrics, except for the “OLD” accuracy of the Imagenet100 dataset. By comparing our proposed method in both Tables 2 and 3, we can observe that our proposed model has a slight decrease in accuracy for the “OLD” class when applying the estimated class number, while the “NEW” accuracy decreases more. This could be due to the prior introduced by the SS-Kmeans, which favours the “OLD” categories more. In addition, we conducted another experiment in Appendix D, Table 7, where we seamlessly adopted the GMM-based dynamic category number estimator into our CCD framework. We would like to emphasize that Table 7 aligns better with our proposed method and is therefore more preferred. As a result, we will swap Table 3 and Table 7 in our final manuscript.

Response to reviewer YA1L

We would like to thank reviewer YA1L for their careful review, insightful feedback, acknowledgement of our model’s novelty and model promising results compared to other baseline models. Here we would like to clarify some concerns addressed by reviewer YA1L:

• Q1: Clarification of our GMP module.

Our Gaussian mixture prompt pool (GMP) differs from L2P and DualPrompt prompt pool in three ways. First, GMP parameters are not fixed as they depend on the number of C components (as shown in Appendix H), which is crucial in CCD tasks where the number of classes in the unlabelled data can grow or reduce over time. Second, we argue that GMM is advantageous over K-means when working with high/complex dimensional data. By using GMM, we allow a sample to have multiple memberships towards different cluster components, i.e., non-convex clusters. K-means assumes that the probability a sample belongs to a cluster is one, which in real-world data might not be true. Third, we show in Appendix I that GMM outperforms K-means for all clustering metrics, which means better prompt quality. Therefore, it is natural to choose GMM as a pool of prompts. For further discussion, please refer to General response - Clarification on L2P & DualPrompt vs Our GMP.

• Q2: Adaptation of L2P and DualPrompt for CCD.

To adapt L2P and DualPrompt as our baseline models for CCD as shown in Section 2.1, main paper, we make the following key modifications: (1) we change the pretrained model from a strong supervised pretrained model to DINO self-supervised pretrained model, (2) Originally in supervised continual settings, L2P and DualPrompt adopt supervised objective loss i.e., cross-entropy loss to optimize their models. For CCD, we modify the loss function into a contrastive learning objective, Section 2.3, the main paper combined with surrogate loss, Eq.(1) to optimize the prompt pool to pull selected keys close to corresponding query features. Please note that for our proposed method i.e., PromptCCD w/GMP, we do not use any surrogate loss as GMM is optimized separately. For details on how we optimize GMM, see Appendix A.

• Q3: Model optimization when the DINO backbone is fully frozen.

In the L2P model, the prompts are learnable parameters which are optimized jointly with the model. Unlike L2P, our prompts are not learnable as it is fitted from the features $z`[CLS]`$ which is learned by fine-tuning the final block of the backbone. Thus, if we fixed the backbone, then the only learnable parameter left in our model is the projection head which is not used during inference i.e. the model becomes fully frozen. Here we provide the comparison results between the frozen DINO w/GMP, frozen DINO w/L2P and our proposed model in Appendix H, Table 13. We can see that our fine-tuned model substantially outperforms the frozen models. Moreover, the frozen models failed to generalize to distribution that DINO had not encountered before (i.e., CIFAR100 and CUB200, as DINO was pre-trained on ImageNet 1K). This led to a significant loss in accuracy compared to the fine-tuned models with average difference losses of 26.43% and 18.29% respectively. This observation further emphasizes the importance of our design choice.

• Q4: Clarification on model’s ablation studies.

Our model’s GMP ablation studies consider three factors: covariance type for GMM, number of prompts, and number of GMM sampling for replay. Table 4 shows that the optimal GMM covariance types for CIFAR100 and CUB200 datasets are “Spherical” and “Full,” respectively, with five prompts and 100 GMM samples. Although there is some ambiguity on the covariance type, we set the default configurations to be “Diagonal” for the GMM covariance type, five prompts, and 100 GMM samples, which appears to be a good trade-off. We further perform an additional ablation study in Appendix K where we keep all covariance of GMMs to be spherical. The results show that 5 prompts and 100 samples lead to the best “ALL” accuracy

Response to reviewer 1nUF

We would like to thank reviewer 1nUF for their careful review, insightful feedback, and acknowledgement for recognising the innovation and novelty of our approach, comprehensive benchmark and extensive experiments, and comprehensive literature review. Here we would like to clarify some concerns addressed by reviewer 1nUF:

• Q1: Suggestion on paper presentations.

We thank reviewer 1nUF for their suggestions. We have revised the manuscript with updated Figures 2 & 3 with added captions and provide the pseudo-code to explain the overall process in Appendix A.

• Q2: Clarifications on how GMP is employed.

We used Gaussian mixture models (GMMs) even during the initial learning with labelled data. The GMM learned the features from the labelled data and transferred this knowledge in the form of GMM random samples and fit them to the next GMM.

• Q3: Discussion on the limitation of GMP.

As shown in Appendix I, Table 13, by learning GMM as a pool of prompt, and finetuned the final layer of the backbone, we observe the performance on C100 and CUB200 datasets to be interesting, we see that our proposed model guided by GMM’s mean components generalized better (compared to frozen DINO backbone) to datasets that the DINO foundation model has not encountered before which further justifies the design choice of our method. This argument is further justified in Appendix H, where we show that GMM’s prompts represent class prototypes. Thus, our method does not limit the model’s capability when the task identity for each stage is different.

Response to reviewer k1JJ

We would like to thank reviewer k1JJ for their careful review, insightful feedback, and acknowledgement of our rigorous analysis of the CCD task, high-level clarity and comprehensive presentation of our paper, and the effectiveness of our proposed method. Here we would like to clarify some concerns addressed by reviewer k1JJ:

• Q1: Clarifications on the difference between the L2P method and our proposed method

In terms of the difference between the L2P method and our proposed method, please refer to General response - Clarification on L2P & DualPrompt vs Our GMP.

• Q2: The motivation for using the self-supervised vision foundation model for the CCD task

We would like to clarify that for L2P, they did not use a self-supervised pretrained model. Instead, they adopt a fully supervised pretrained model on very large-scale labelled data i.e., ImageNet-21K. In contrast, we use the fully self-supervised pretrained model DINO, which has been widely adopted in the static GCD task, by leveraging its strong feature generalization capability. DINO is a self-supervised (SSL) pretrained model (no labels are used during training). We want to point out that the SSL model has been widely adopted and justified in both NCD [A] and all GCD literature so far. It is a valid and reasonable choice to use such models because no supervision, such as class labels, was used to train the feature backbone.

• Q3: Information on the learnable parameters from all compared models.

We provide the table comparison of each model’s learnable parameters in Appendix H. Here we show that our model’s GMM parameters are flexible as we only need to define the number of components to scale it. Overall, compared to our baseline models (L2P and DualPrompt), Our proposed model’s learnable parameters are only 0.33% higher with category number C set to 100, which is still parameter efficient.

• Q4: Fixing the number of categories for main experiments.

Yes, in the main table experiments i.e., Table 2, we fix the number of categories to a predefined value for all models and datasets for fair comparison.

• Q5: Comparison between the predicted number of categories vs the actual number of categories.

We provide the tables for comparison between the predicted number of categories and the ground truth categories in Table 3 (GCD [B] category number estimator) and Appendix D (Ours, by seamlessly adopting the GMM based dynamic category number estimator from GPC [C]). Overall, in this realistic scenario, our model still consistently outperforms other methods by a large margin across different datasets.

• Q6: The feasibility of extending the number of discovery stages.

For a fair comparison, we follow the previous work [D] data distribution i.e., 3 discovery stages. Also, we would like to clarify that our approach does not have any constraint on the number of discovery stages. In Appendix E we further compare our model with the recent ICCV-accepted papers called MetaGCD with 4 incremental category discovery stages and PAiGCD with 2 incremental category discovery stages.

General response section, part 3

Reference:

[A] Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Autonovel: Automatically discovering and learning novel visual categories. IEEE TPAMI, 2021.

[B] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In CVPR, 2022.

[C] Bingchen Zhao, Xin Wen, and Kai Han. Learning semi-supervised gaussian mixture models for generalized category discovery. In ICCV, 2023.

[D] Xinwei Zhang, Jianwen Jiang, Yutong Feng, Zhi-Fan Wu, Xibin Zhao, Hai Wan, Mingqian Tang, Rong Jin, and Yue Gao. Grow and merge: A unified framework for continuous categories discovery. In NeurIPS, 2022.

[E] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR, 2022b.

[F] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In ECCV, 2022a.

General response section, part 2

2. Clarification on L2P & DualPrompt vs Our GMP (R-k1JJ, R-YA1L, R-si3G)

Our motivation is to design a prompt module that effectively guides the self-supervised vision foundation model in open-world continual category discovery. L2P [E] and DualPrompt [F] are two methods developed for supervised continual learning that utilizes a prompt pool to guide a strong supervised pretrained model during learning. Therefore, They are not applicable for the CCD task. However, as we show in Section 2.1, the main paper and R-YA1L-Q2, these models can be carefully modified for CCD with the necessary changes, despite being less effective than our method. However, there are a few critical limitations for these solutions. First, the representation learning only takes into account the unlabelled data in the current time step, which may lead to representation bias towards current data and disrupt the representation learned for the previous data. Second, the category discovery process is disjoint from the representation learning, therefore, lacking a proper mechanism for transferring knowledge from old classes to new classes, which is essential for the category discovery task. To address this issue, we proposed the novel GMM-inspired prompt called GMP (the details on the proposed method and the overall training process can be seen in Section 2.2, main paper and Appendix A respectively), Our proposed model introduces three significant improvements compared to existing models when tackling CCD (R-k1JJ-Q1, R-YA1L-Q1, R-si3G-Q2). Firstly, GMP’s prompt serves a dual role, namely (1) as task prompts to instruct the model (like in L2P and DualPrompt) and (2) as class prototypes (as shown in Appendix I) to act as parametric replay sample distribution for discovered classes. The second role, which is unique and important for CCD/GCD, not only allows the model to draw unlimited replay samples to facilitate the representation tuning and class discovery in the next time step but also allows the model to transfer knowledge of previously discovered and novel categories and incorporate this information when making the decision to discover a novel category (R-si3G-Q1). Moreover, there are other limitations of L2P and DualPrompt when adapted to CCD. First, their fixed-size prompt modules can lead to parameter inefficiency and restrict the model’s ability to discover new categories and avoid forgetting. GMP can easily overcome this as the GMP module allows easy parameter adjustment and prevents forgetting by sampling and transferring samples to the next GMM at each time step. By transferring these random samples from the previous stage into the current GMM, we ensure that the learned components from the previous stage are retained and carried forward. This process of transferring the components effectively reduces the risk of forgetting previously learned information, enhancing the model's ability to preserve knowledge and adapt to new categories. Lastly, compared to L2P and DualPrompt, GMP is more adaptable to CCD tasks as it can be seamlessly combined with the dynamic GMM-based category number estimator [C] without introducing any extra modules or parameters (See Appendix D for details and experiment results). Thus, by incorporating this dynamic category estimation without introducing any extra modules or parameters, our model becomes more adaptive and capable of handling the challenges posed by an ever-changing open world which allows our approach to effectively discover and accommodate new categories as they emerge, making it a valuable solution for real-world applications. Overall, our method is a highly optimized framework specialized in addressing the challenges in the CCD task, which can not be handled by L2P and DualPrompt.

*Note that for supervised continual learning models such as L2P and DualPrompt adopt a strong supervised vision foundation model with incorporate label information during training.