Happy: A Debiased Learning Framework for Continual Generalized Category Discovery

Thank you for your insightful advice and valuable questions, we will respond to your concerns point by point.

W1: Complexity of the Framework.

• Effect of each component. Considering that C-GCD is a challenging task, each component is essential and addresses specific issues. (1) Learning new class: Cluster-guided initialization provides a robust initialization for new class heads, while entropy regularization mitigates prediction bias and allocates necessary probabilities for discovering new classes. (2) Preventing forgetting old classes: hardness-aware prototype sampling effectively controls the forgetting of difficult old classes.

• Implementation. (1) Cluster-guided only runs once at each continual stage. (2) Soft entropy regularization, (3) hardness modeling could be implemented on the fly. All of them consume less than 1% of total memory and time, see W3 for more details.

• Hyper-parameter tuning.

  • We fix the loss weights of self-training $L_{self-train}$ and hardness-aware prototype sampling $L_{hap}$ as 1, considering they are the main loss for new and old classes. And our method basically has three weight parameters $\lambda_1,\lambda_2,\lambda_3$ for $L_{entropy-reg}$, $L_{kd}$ and $L_{con}^u$ respectively. Please see General Response to Common Concern 1 for more details and sensitivity analysis.
  • Overall, the optimal values for each hyperparameter are close to 1. In our experiments, we simply set all weights to 1, which shows remarkable results across all datasets. Thus, our method does not require complex tuning of parameters and exhibits strong generalization capabilities.

W2: Scalability of the proposed method.

• The time consumption increases linearly with the number of classes and stages, which is the same for all continual learning methods.
• Disk Memory consumption. To mitigate forgetting of old classes, we save one prototype feature per class (768-dim) using about 3KB, whereas previous methods store multiple labeled images, with one image 224× requiring 588KB and ten images requiring over 5000KB. Thus, our method consumes less than 1/1000 of previous replay-based methods.

W3: Evaluation Metrics: computational efficiency, memory usage, or robustness to different distributions.

• Computational efficiency. The proposed three components are computationally efficient. We run experiments on a single 4090 GPU. For CIFAR100, we train 5 continual stages with 30 epochs for each, it takes $\sim$2.5h in total. (1) Cluster-guided initialization is run only once at the start of each stage, it takes $\sim$15s, only 0.83% of total time. (2) Hardness-aware sampling and (3) entropy regularization occupy less than 1% of the training time. While KD loss takes relatively the most time, around 30% of the time, however, it is necessary and widely used in continual learning.
• Disk memory consumption. As in W2, we only save features for old classes with only <1/1000 memory cost compared with replay-based continual learning. (3KB/5000KB) and our method alleviates privacy issues.
• GPU memory consumption. Training costs around 14192MB GPU memory in total. Within it, hardness-aware sampling and entropy regularization occupy less than 0.1%, and KD loss cost 5672MB, whose main objective is to mitigate forgetting, it consumes even less memory than replay-based methods ($\sim$8000MB) with the same batch size.
• Robustness to different distributions. We have evaluated various methods on several corrupted distributions, see General Response to Common Concern 2.

Q1: Generalizability to Other Domains.

• Good question. This paper primarily explores visual tasks of visual category discovery. The underlying spirit could also be adapted to NLP data, we leave it for future work.
• For other distributions, we conduct experiments on CIFAR100-C with various corrupted distribution, see General Response to Common Concern 2.

Q2: Adapt to the SoTA methods.

• Great suggestion. We implemented SPTNet [R1] and adapted it to the C-GCD (by incorporating alternate prompt learning), results on CIFAR100 are as follows:

  CIFAR100	All	Old	New
  SPTNet	59.54	67.84	8.56
  SPTNet+LwF	63.89	70.33	24.31
  Happy	69.00	71.82	51.36
  SPTNet+Happy	70.81	73.58	54.18

  we also adapt InfoSieve [R2] (by incorporating self-supervised code extraction into contrastive learning) as you suggested, results on CUB are shown below:

  CUB	All	Old	New
  InfoSieve	57.23	65.28	7.75
  InfoSieve+LwF	63.18	69.30	25.60
  Happy	68.88	71.29	53.13
  InfoSieve +Happy	69.80	72.32	54.30

  Results indicate that adapting SoTA GCD methods to C-GCD with Happy could further enhance the performance.

• We will cite related papers and add them to the revised manuscript.

References:

[R1]. SPTNet: An Efficient Alternative Framework for Generalized Category Discovery with Spatial Prompt Tuning. ICLR 2024.

[R2]. Learn to Categorize or Categorize to Learn? Self-Coding for Generalized Category Discovery. NeurIPS 2023.

Dear Reviewer cVL6,

Thanks for your feedback. Considering the "weaknesses of the other Reviewers", we have given responses point by point in the rebuttal. Here, we summarize our responses to the weaknesses pointed out by all reviewers:

• Hyper-parameters tuning. Please refer to General Response to Common Concern 1 and response to you in W1. Our method requires three hyper-parameters, and we only need to set all weights to 1, which is generalizable and could perform well across all datasets. Therefore, the parameter tuning is simple and easy to implement.

• Complexity of the method.

  • Firstly, we have demonstrated the necessity of each module, as shown in Table 5 of the main manuscript.
  • Secondly, we have provided the computational overhead for our proposed modules (see responses to you in W2 and W3). The cluster-guided initialization, entropy regularization, and hardness modeling that we propose consume less than 1% of time and memory resources, which means that they are very efficient and effective.

• Scalability of the method. All continual learning methods experience a linear increase in computational demand with the number of classes and stages. Compared to previous replay-based methods, our approach saves disk storage by a factor of 1/1000, as detailed in the response to you in W2.

• Generalization abilities of the method.

  • We have conducted evaluations on 15 corrupted and shifted distributions of CIFAR100-C, as in the General Response to Common Concern 1 and Rebuttal PDF.

  • Furthermore, previously, we have also conducted experiments on two more fine-grained datasets: Stanford Cars and FGVC Aircraft datasets, with the results as follows:

    Standford Cars	All	Old	New
    GCD	47.00	47.73	42.61
    MetaGCD	54.67	55.28	50.95
    Happy (Ours)	62.79	63.68	57.34

    FGVC Aircraft	All	Old	New
    GCD	42.95	44.35	33.38
    MetaGCD	47.16	48.61	38.23
    Happy (Ours)	53.10	53.81	48.71

  • To conclude, our method achieves remarkable performance compared to the previous sota on 6 datasets and 15 unseen distributions, showcasing the strong generalization abilities of Happy.

• Adapting to SoTA methods. We have adapted our method to two sota methods: SPTNet and InfoSieve. Please refer to response to you in Q2 for more details. We will add all results of adapting Happy to all three methods you listed in the final version. We will also add citations of the three methods: SPTNet TIDA, and InfoSieve in our reference list.

Thank you again for your feedback. We hope our responses and extensive experiments have addressed your concerns. If you have any further questions, we welcome continued discussion. Thank you once again for your review.

Thank you for your insightful advice and valuable questions, we will respond to your concerns point by point.

W1: Why not use these old class data directly instead of hardness-aware prototype sampling?

• Good question. At each continual stage of C-GCD, all training data are unlabeled, i.e., unlabeled old and unlabeled new classes samples are mixed together, and the model does not know which samples belong to old categories, thus they cannot be directly utilized.
• We employ hardness-aware prototype sampling to prevent forgetting, without the need to store class-wise labeled samples of old classes. By contrast, previous methods store class-wise labeled data, which could cause privacy issues.

W2: How to compute margin prob in line 173 without labels?

• For the marginal probability, we compute the average of model predictive probabilities across a batch, namely $\overline p\in\mathbb{R}^{K^t}=\frac{1}{|B|}\sum_{i\in B}p_i$, which does not require any labels, and only uses model predictions.

W3: DINO pretrained ViT-B/16 backbone results in information leakage for CIFAR100 and TinyImageNet.

• Very good question. We will address this point in detail and incorporate them into the revision.

• Firstly, DINO is an self-supervised (unsupervised) pre-training scheme without any labels. So there is no label leakage in DINO. The main objective of DINO is to learn good and general initial representations for the visual encoder. DINO does not directly learn classification heads for downstream tasks.

• Second, since using a DINO-pretrained ViT has become a standard practice in the literature of GCD [R1, R2, R3], we just followed this setup to ensure a fair comparison.

• Third, we acknowledge your concerns. To thoroughly eliminate the influence of information leakage, we adopted the settings in [R4, R5] and conducted experiments using a DeiT pretrained on 611 ImageNet categories (explicitly excluding those from CIFAR and TinyImageNet). Average accuracy over 5 continual stages are shown below:

  		CIFAR100			TinyImageNet
  	All	Old	New	All	Old	New
  VanillaGCD	42.50	44.57	28.16	29.28	30.04	24.02
  MetaGCD	46.64	48.63	32.96	35.24	36.48	26.80
  Happy(Ours)	69.34	71.37	57.44	53.50	55.67	39.40

  As it shows, in scenarios without any information leakage, our method still consistently outperforms previous methods by a large margin. In contrast, methods like MetaGCD, which rely on non-parametric clustering, suffer significant performance declines due to unstable clustering with less pretrained classes.

References:

[R1]. Generalized Category Discovery. CVPR 2022.

[R2]. Parametric Classification for Generalized Category Discovery: A Baseline Study. ICCV 2023.

[R3]. MetaGCD: Learning to Continually Learn in Generalized Category Discovery. ICCV 2023.

[R4]. Learnability and algorithm for continual learning. ICML 2023.

[R5]. Class Incremental Learning via Likelihood Ratio Based Task Prediction. ICLR 2024.

Thank you for your insightful advice and valuable questions, we will respond to your concerns point by point.

W1: Overclaim the CGCD setup.

• Differences from [16,18]. We consider (1) more continual stages (5>3 and 10>3 in table 4) with more novel classes to discover (50%>30% of total classes are new) (2) rehearsal-free, i.e., we do not save class-wise labeled samples for old classes, which alleviate privacy and memory issues.
• Thanks for pointing out the issue. We acknowledge that the settings of our study still have some gaps compared to real-world scenarios and it is a basic exploration towards real-world applications.
• We will revise the statement as you suggested.

W2: Too many hyper-parameters and loss terms.

• Explain of each loss function. Overall, our method can be divided into three parts: $L_{Happy}=L_{new}+L_{old}+\lambda_3L_{con}^u$. Each of them is important, see table 5 for ablation studies.

  • $L_{new}$: discover new classes (cluster). It has two parts (1) self-distillation $L_{self-train}$ with sharpen targets for self-training. (2) soft entropy regularization $L_{entropy-reg}$ to alleviate prediction bias, wherein eq (4) is regularization between old and new classes as a whole.
  • $L_{old}$: mitigating forgetting of old classes. It also has two parts (1) hardness-aware prototype sampling $L_{hap}$ eq (10) and (2) KD loss $L_{kd}$.
  • $L^u_{con}$: unsupervised contrastive learning (i.e., SimCLR) for basic representation.

• The effect of eq (4) is to reserve the necessary probability for learning new classes and mitigate prediction bias towards old classes.

• About hyper-parameters.

  • In our method, we fix the weight of $L_{self-train}$ and $L_{hap}$ as 1, considering they are related to the core part of learning new and old classes. (just like cross-entropy loss in standard deep learning)
  • We only need to tune the weights $\lambda_1,\lambda_2,\lambda_3$ for $L_{entropy-reg}$, $L_{kd}$ and $L_{con}^u$, respectively. For the detailed sensitivity analysis, see the table in General Response to Common Concern 1. The optimal weight for the three loss are all near to 1.
  • In practice, we set all the loss weights to 1 for all datasets without fussy tuning, which works fine and generalizes well to several datasets. As a result, we do not need to extensive search, just set all the loss weights to 1 and it works fine.

W3: In table 5, what does new class mean? Why KD improve new class by a large margin?

• Good question. In table 5, new class means the average accuracy of new classes over 5 stages, i.e., $C_{new}^1,\cdots, C_{new}^5$. i.e., we report the average of new accuracy across five continual tasks.
• In C-GCD, the classification performance is determined by both the feature space and the classification head.
  • Without KD loss, the feature space may shift and become mismatched with the head, causing the model to misclassify some new classes as old, which significantly reduces the accuracy of new classes.
  • Additionally, since the evaluation of C-GCD relies on Hungarian matching across all classes to achieve the highest overall accuracy (see line 224), there are generally more old classes, which tends to stabilize the old classes while new classes experience greater fluctuation.

W4: Provide evidence to show that two issues are mitigated.

• Very good suggestion. We will append the following results to the revised manuscript.

• Prediction bias. We provide two metrics: (1) $\Delta p=\overline p_{old}-\overline{p}_{new}$: the difference in marginal probabilities between old and new classes. (2) the proportion of new classes' samples misclassified as old classes (new$\to$old). The results are as follows (after stage-1):

  (in %)	$\Delta p \downarrow$ (on C100)	new$\to$old $\downarrow$ (on C100)	$\Delta p \downarrow$ (on CUB)	new$\to$old $\downarrow$ (on CUB)
  Ours w/o $L_{entropy-reg}$	81.50	63.25	83.20	65.80
  Ours w/ $L_{entropy-reg}$	5.76	10.20	10.25	11.05

  The results from two datasets demonstrate that $L_{entropy-reg}$ effectively reduces prediction bias, with a significantly lower marginal probability gap and fewer new class samples misclassified as the old.

• Hardness bias. We also present two metrics: (1) $Var_0$: variance in accuracy of the initial labeled classes $C_{init}^0$. (2) hardest Acc: accuracy of the hardest classes in $C_{init}^0$. Results are as follows (after training on 5 stages):

  (in %)	$Var_0 \downarrow$ (on C100)	hardest Acc $\uparrow$ (on C100)	$Var_0 \downarrow$ (on CUB)	hardest Acc $\uparrow$ (on CUB)
  Ours w/o hardness sampling	23.04	65.10	21.77	62.65
  Ours w/ hardness sampling	10.33	70.23	9.28	68.40

  Results show that hardness-aware sampling effectively reduces hardness bias, with lower accuracy variance and higher hardest Acc.

Minor typos.

• Thanks for your detailed review. We will correct all the typos carefully.

Thank you for your insightful advice and valuable questions, we will respond to your concerns point by point.

W1: Lack of a comprehensive social impact analysis.

• We summarize the negative impacts: (1) Bias/error accumulation. Given the stringent unlabeled conditions of C-GCD, the initial learning bias may continuously accumulate, leading to a greater bias in later stages. (2) Spurious correlations. Category discovery relies on knowledge learned from labeled classes. If incorrect correlations are learned, e.g., over-reliance on the background, these may interfere with the learning of new ones.

• We will discuss them in detail in the final version.

W2: Lack of a theoretical foundation.

• Happy is essentially self-learning on unlabeled data, and the theoretical foundation is conditional entropy maximization.

• Specifically, the objective can be expressed as $\max I(Y, Z) = H(Y) - H(Y|Z)$, where $Y$ represents the labels and $Z$ the features, with $I$ and $H$ denoting mutual information and entropy, respectively. The first term is equivalent to the proposed marginal entropy regularization $L_{entropy-reg}$, while the second one $H(Y∣Z)$, can be computed by the posterior probability $p(y|z)$, which we aim to minimize through self-distillation $L_{self-train}$.

• We will include this theoretical analysis.

W3: There are assumptions that may limit the general applicability especially in different distributions.

• Our method achieves sota not only on four datasets (table 2) but also on the distribution-shift CIFAR100-C. (See General Common Concern 2 for more details). The results demonstrate that our method has a stronger generalization ability.

W4: The accuracy of the new class still fluctuates at different incremental stages with the debias methods.

• The fluctuation in new categories is an inherent issue in C-GCD. However, our method exhibits significantly less variability compared to others, e.g., on CIFAR100, the maximum new accuracy difference across stages is $\Delta_{new}=6.3$ (56.1-49.8), which is less than the 17.3 (48.9-31.6) of MetaGCD. Similar patterns can be observed in other datasets as well.

Q1: How to make sure the model not only finds new categories but also avoids confusing them with the old ones? Are there specific metrics?

• We follow SimGCD and employ a unified classification head for both old and new classes in a joint prediction space. So our method could classify both new and old categories together, which not only implicitly differentiates between new and old classes but also discovers new ones.

• Specific metrics. We calculate the proportion of new class samples misclassified as old classes (new$\to$old) and old classes misclassified as new (old$\to$new) after stage-1 training. Results are shown below:

  misclassify ratio	new$\to$old (C100)	old$\to$new (C100)	new$\to$old (CUB)	old$\to$new (CUB)
  SimGCD	85.75	6.78	84.18	6.65
  MetaGCD	30.65	6.64	32.36	6.50
  Happy	10.20	4.35	11.05	4.95

  Results show Happy can more effectively distinguish between new and old classes with a lower misclassify ratio, and achieves a higher accuracy for new classes (e.g., 51.3>31.6 at stage-5 in table 2).

Q2: Explain soft entropy regularization and cluster-guided initialization.

• Soft entropy regularization. See Sec 3.2 line 128, as the model tends to bias the predictions to old classes, so we implement an explicit constraint to ensure necessary predictive probabilities for new classes to facilitate learning. Considering that at each stage the prior ratio of old and new classes is inaccessible, we employ a soft regularization, assuming a general balance between the probabilities for new and old classes ($p_{old} = p_{new} = 0.5$), which is achieved by the marginal entropy maximization.

• Cluster-guided initialization ensures a good initialization for new class heads, while entropy regularization improves training of new classes with less prediction bias.

Q3: How to define "hardness" and how to sample against this hardness? Concrete examples of how it helps remember difficult old classes?

• As in Sec 3.4 Eq (9), we define class-wise hardness $h_c$ as the average cosine similarity between the class center $\mu_c$ and other class center $\mu_j$, where $\mu_j$ is computed in Eq (8). Intuitively, the greater the similarity to other classes, the more likely it is that its feature space will be confused with others, thereby the class is more difficult.

• To sample against hardness, considering more difficult categories tend to be forgotten more readily, we sample harder classes more frequently. So we treat $h_c$ as logits and model hardness distribution with softmax function in Eq (9).

• Concrete samples. After learning 5 stages, we evaluate the model on initial labeled classes $C_{init}^0$ and the accuracy of the hardest class (lowest acc) is reported:

  hardest Acc	C100	CUB
  SimGCD	62.85	59.68
  MetaGCD	65.31	61.26
  Happy	70.23	68.40

  Results show that Happy could remember difficult classes better with higher hard class accuracy.

Q4: Have other technologies been considered to further improve the task?

• MetaGCD actually utilizes meta-learning, but the results are weaker than ours (see table 2), because MetaGCD only simulates the C-GCD process without any debiasing mechanism.
• In essence, meta-learning and memory enhancement can be integrated with our debiasing mechanism to further enhance the results. We leave this aspect to future work.