Continual Generalized Category Discovery: Learning and Forgetting from a Bayesian Perspective

Claims And Evidence:

The constrained optimization equation formally represents the objective of "learning new classes without forgetting old ones." We observed that maximizing overall accuracy inevitably leads to a decline in the classification performance of previous classes, as illustrated in P2 of Figure 2. Therefore, considering the interplay between old and new classes, we believe that this issue is a multi-objective optimization problem. Accordingly, we relax the original constraint by imposing a covariance constraint, which approximates a balance between preventing forgetting and effectively learning new classes.

With advancements in self-supervised pre-training, the effectiveness of self-supervised fine-tuning in small-scale scenarios has diminished. On the other hand, due to the presence of classification errors, employing supervised fine-tuning during online sessions can introduce a significant number of misclassifications, which in turn leads to a decline in overall performance. Meanwhile, we did not entirely freeze the backbone; instead, we performed fine-tuning with labeled data during the offline phase, which was performed using verified ground-truth labels.

Most methods use the MVN as a sampling distribution for data replay or to assist clustering, rather than directly as a classification prototype. In contrast, VB-CGCD approximates the MVN through variational inference, and we believe that this strategy is the key to our improved performance.

Experimental Designs Or Analyses:

We acknowledge that Variational Inference can be computationally intensive. However, our VI process does not need to reach full convergence; only a few steps (1000 epochs in our experiments) are sufficient to distinguish different categories. Additionally, by not requiring fine-tuning of the backbone network during training, we only need to extract features once for each sample, performing a single forward pass through the backbone, which significantly reduces training time. Consequently, compared to other methods that necessitate frequent fine-tuning of the backbone network, our training time is substantially reduced. The running time during training on one RTX-8000 is shown as follows:

Compared to HAPPY (over 25 hours of training) and PromptCCD (over 40 hours of training), VB-CGCD demonstrates significantly superior training efficiency. After training, VB-CGCD achieves inference computational efficiency comparable to other prototype-based approaches, normally in milliseconds.

Q&A

As a dynamically extensible incremental learning classifier, VB-CGCD can be integrated with advanced clustering algorithms, including those capable of dynamically estimating the number of categories. Since VB-CGCD does not require a preset number of categories, it can be combined with many off-the-shelf methods for estimating unknown category counts. We employed the classical silhouette score to implement a method for estimating the number of unknown categories and integrated it into VB-CGCD. For experimental results, please refer to Response 2 of Reviewer 1.

VB-CGCD models each category as an independent probability distribution and maintains a collection of all category distributions (i.e., a matrix of latent variables) in memory. After estimating the pseudo-label distribution, we incorporate it into the distribution collection (prototype matrix). During re-labeling, the distances between all samples and this collection are computed to assign labels accordingly. Subsequently, the pseudo-distribution is removed from the collection, and a new distribution is re-estimated. The merge operation is purely an array manipulation, which accounts for VB-CGCD’s inherent scalability, and the reason for being able to handle unknown number of categories.

Since the primary goal of continual learning is to enhance overall learning performance, we agree that preventing overfitting to new classes is necessary to mitigate the forgetting of previously acquired knowledge. In fact, many continual learning methods, whether by adding regularization constraints or employing data replay, aim to some extent to slow down the learning of new classes to prevent forgetting old ones. Without imposing any constraints and allowing unrestricted learning of new classes, the model's classification accuracy on new classes may continue to improve, but the accuracy on old classes could drop drastically, as illustrated in phase P3 of Figure 2. Therefore, to maximize overall accuracy, continual learning must strike a balance between stability (preventing forgetting) and plasticity (learning new categories). This trade-off inherently implies that optimizing one aspect will inevitably come at the expense of the other.

Weakness:

We acknowledge that DINO-ViT-B16 was pre-trained on the ImageNet dataset, though its training was based on self-supervised learning using unlabeled data, and it was not trained on CIFAR or CUB. Since all our baselines including HAPPY (SOTA) utilize DINO for feature extraction, VB-CGCD also uses DINO to ensure a fair comparison with other methods. Additionally, to further assess the generalizability of our approach, we have conducted experiments on Stanford Cars, FGVC-Aircraft, CORE50, and Food-101 datasets.

Compared to other baselines, VB-CGCD outperforms them by a decent margin in terms of average overall accuracy, as shown below:

Thank you for your valuable feedback. We will include a discussion on FRoST (Class-iNCD) in the related work section. FRoST employs a Gaussian distribution as the sampling distribution for its replay mechanism, incorporating samples from previously learned classes during training to mitigate forgetting. This represents a typical replay-based approach. In contrast, VB-CGCD does not utilize a replay mechanism but instead relies directly on prototypes for classification. The experimental comparison between FRoST and VB-CGCD is as follows:

In terms of final overall accuracy, VB-CGCD outperforms FRoST by an average of 30.76%, demonstrating its superior performance.

In unsupervised scenarios, most methods are influenced by the performance of the clustering algorithm. Our strategy is to minimize errors introduced by the clustering algorithm from the classifier's perspective to enhance performance. To ensure fair comparisons with other methods, we have employed the fundamental k-means algorithm. Of course, our approach can also be integrated with more advanced clustering algorithms to further improve performance.

Q&A

Across all datasets and tasks, we employed a consistent set of hyperparameters, including learning rate and training epochs. Specifically, the fine-tuning coefficient λ was uniformly set to 0.001. Regarding the early stopping strategy, we posit that equalizing the average covariance between new and old classes effectively mitigates class bias. Therefore, we monitor the covariance of both new and old classes during training and halt training when they are equal, i.e., when R=0. We have conducted a sensitivity analysis on the R. Please refer to Response 4 of Reviewer 2.

Since our method performs density estimation solely on features, it can be broadly applied to various modalities. We conducted experiments on audio and text classification tasks.
For audio we evaluated our method on esc50, speechcommands, and audiomnist datasets. For text, we evaluated stackoverflow, and clinc.

The experimental results indicate that VB-CGCD demonstrates good performance in these tasks as well, further validating its generalizability and robustness to other modalities.

We appreciate the reviewer’s valuable feedback. We have regarded HAPPY as an important SOTA but inadvertently missed its discussion in the related work section. We will include the discussion of HAPPY in the revised version, with a particular emphasis on its technique of using Gaussian distributions for prototype sampling, to provide a more comprehensive overview of the relevant work.

We reproduced the experimental results of HAPPY and found that our reproduced results outperform those reported in the original paper. We reported the reproduced results in our paper.

We will update the overall process in the caption of Table 1 as follows:
"(a) Offline Stage: Labeled data is used to fine-tune the backbone network. After fine-tuning, features are extracted, and variational inference is performed to obtain and store the class distribution.
(b) Online Stage: Unlabeled data is first processed by the feature network to extract features, which are then clustered to assign pseudo-labels. These pseudo-labels are combined with a first round of variational inference to obtain an approximate distribution, then combined with the stored class distribution from the previous session for re-labeling."

Across all datasets, we employed the same hyperparameter configuration, including conventional parameters such as epochs and learning rate, as detailed in Appendix C.3.
Regarding the fine-tuning hyperparameter λ, we found that the contribution of self-supervision to performance is almost negligible. This is primarily because, compared to self-supervised learning on extremely large-scale pretraining datasets, the fine-tuning dataset has a limited impact on performance, with the cross-entropy loss playing a more crucial role. Therefore, we set λ to a very small value of 0.001.

As for the early stopping hyperparameter, as discussed in Figure 2, we argue that ensuring equal average covariance between new and old classes can effectively mitigate class bias. Consequently, our early stopping strategy dictates training to be stopped once the covariances of new and old classes are equal (i.e., when R equals 0).
We conducted experiments on the CIFAR 100 dataset, focusing on the parameter R, adjusting its value to achieve different trade-offs between model plasticity and stability.

Comparisons in overall accuracy as R varies

Comparisons in the accuracy of old classes as R varies

Comparisons in the accuracy of new classes as R varies

The experimental results indicate that the parameter R serves to balance the accuracy between new and old classes, with the overall accuracy being highest when R equals 0. We will include these details in the updated version of the paper.

In continual learning, new data is introduced at each session. If the backbone network is continuously fine-tuned, the features extracted from the same input using different versions of the network (e.g., h₁ and h₂) will inevitably differ, i.e., h₁(x) ≠ h₂(x). We designed an experiment on CIFAR-100, where we continuously perform self-supervised fine-tuning using data from each session. Subsequently, we evaluate feature drift by conducting supervised learning.

The experimental results indicate that as fine-tuning progresses, the accuracy of each session, as well as the overall accuracy, fluctuates within a 2% range, demonstrating the presence of feature drift. Moreover, this drift does not have a consistently positive impact on accuracy and does not lead to significant performance improvements. Since our classification relies on storing the distribution of old classes, any drift may result in inaccurate classifications. Therefore, we chose to avoid continuously fine-tuning the pre-trained network.

We conducted an ablation study on the benefit of introducing Mahalanobis distance, and the results obtained by replacing the Mahalanobis with the Euclidean distance are as follows:

Compared to VB-CGCD with Mahalanobis distance, the overall final accuracy decreased by approximately 5.56 on average. This is because Euclidean distance, being a special case of Mahalanobis, is susceptible to collapse in high-dimensional spaces. By incorporating covariance information, Mahalanobis distance effectively mitigates this issue, thereby achieving a more optimal classification boundary. We appreciate your valuable feedback and will incorporate this aspect into the revised version of our paper.

Since our proposed approach primarily serves as an incremental learning classifier and is decoupled from the clustering module, it allows integration with any clustering method (including those that estimate the number of categories).
As mentioned in the paper, there are many off-the-shelf methods [1][2] available for estimating the number of categories.
Nonetheless, we employed the classic silhouette score to implement a method for estimating the number of unknown classes, which we integrated into VB-CGCD.
The results are as follows:

Due to errors in estimating the number of unknown classes, the clustering error is exacerbated, ultimately leading to an average overall accuracy reduction of approximately 5.66. Nonethless, it demonstrates the scalability of our method to handle scenarios with an unknown number of types, and our performance still outperforms the SOTA method, HAPPY, which also uses the silhouette score.

[1] DeepDPM: Deep Clustering with an Unknown Number of Clusters. Meitar Ronen, Shahaf E. Finder, Oren Freifeld. CVPR 2022.
[2] PromptCCD: Learning Gaussian Mixture Prompt Pool for Continual Category Discovery. Fernando Julio Cendra, Bingchen Zhao, Kai Han. ECCV 2024.

Our approach distinguishes between known and unknown classes by approximately estimating the unknown distribution, thereby effectively reducing the interference of known classes on the learning of new ones—a strategy that is similar to certain open-set methods. However, there are two key differences between GCD and CGCD: 1) GCD does not need to address the forgetting problem inherent in continual learning; 2) In the unsupervised learning phase of CGCD, supervised data is inaccessible, making it unsuitable for semi-supervised approaches. For example, during training, DPN relies on using L_{known} as a part of the loss function, which renders it inapplicable in CGCD scenarios. Moreover, methodologically, DPN uses the mean as its prototype; however, in high-dimensional spaces, this approach is prone to feature collapse. In contrast, utilizing a Gaussian distribution is more robust. We will further emphasize this distinction in the related work section.

Additionally, we evaluated VB-CGCD on the same datasets which were evaluated in DPN ‘s paper, and the results are shown below.

The experimental results indicate that VB-CGCD achieves performance comparable to DPN. However, it is important to note that VB-CGCD does not need access to any labeled data during the unsupervised learning phase, which is a distinct difference from DPN.