HiLo: A Learning Framework for Generalized Category Discovery Robust to Domain Shifts

The novelty of the method appears limited, as it seems to combine various techniques from different domains.

Please see the General Response for our response regarding novelty of the method.

The comparison with UniOT should be included in the main results. Since the proposed setting is similar to universal domain adaptation, it is essential to compare methods from both domains in the main results.

Following the suggestion, we have conducted experiments using UniOT on SSB-C. The results are presented in Table S2 and also incorporated into Table 3 in the main paper. Our HiLo continues to outperform all other methods.

Table S2: Evaluation on SSB-C. Bold values represent the best results.

Missing citation for the following important paper

We have included these two works in our related work following the suggestion.

The author needs to differentiate between the GCD task setting and the domain shift GCD task setting, so this statement should be revised for clarity and precision.

Following the suggestion, we have revised the corresponding sentence in Line 164-167 to “The objective of GCD with domain shifts is to classify all unlabelled images in $\mathcal{D}^u$ (from either $\Omega^a$ or $\Omega^b$) using only the labels in $\mathcal{D}^l$. This is different from the setting of NCD with domain shift and GCD, which assumes $\mathcal{Y}^l\cap\mathcal{Y}^u=\emptyset$ for the former and $\Omega^a=\Omega^b$ with $|\Omega^a|=|\Omega^b|=1$ for the latter.”

The font sizes of the tables are not standardized, and the font in table 2 is too small.

Thanks for the suggestion. We will properly fix it in our final version. Specifically, we will report the average results across different domains in the main paper and move the breakdown evaluation of each domain shift to Appendix.

The authors should have listed the error lines generated by the three independent runs.

All results in Table 2 and 3 are averaged by three trials with different random seeds. Following your suggestion, we have added the bar chars with corresponding error bars for different methods across DomainNet and SSB-C in Appendix O.

For the experimental results of the ORCA method, what is the backbone used by the authors?

We use the pretrained DINO model as the backbone for all methods to ensure fair comparison, which is a common practice in the GCD literature (Wen et al. (2023); Wang et al. (2024)).

Were any curriculum learning alternatives considered, such as adaptive weighting based on difficulty or dynamic sample weighting? A brief discussion on these choices would clarify why the current approach was favored.

Indeed, there are several common curriculum learning strategies: (1) difficulty-based adaptive weighting, which adjusts sample weights based on model performance [S9]; (2) dynamic sample weighting, which updates weights during training based on learning progress [S10]. Our method aligns with these approaches in principle, as domain shift naturally correlates with learning difficulty - samples with larger shifts from the source domain are inherently more challenging to learn from. However, rather than computing difficulty measures during training, we pre-compute sampling weights based on domain shifts before training begins. This design choice offers two key benefits: (1) computational efficiency by avoiding per-iteration difficulty assessment, and (2) more stable training by preventing potential oscillations in difficulty estimates that can occur with dynamic weighting. This approach captures similar intuitions about progressive learning.

To address this comment, we further explored one of each type, CL [S8] and Self-Paced Learning (SPL) [S9]. For CL, we implemented it by gradually including more difficult samples based on classification loss, starting with 20% of the easiest samples and increasing by 10% every 20 epochs. We also integrated SPL into our framework by adding a weighted loss term $\lambda||v||1 + \sum{i}v_i\ell_i$, where $v_i$ indicates sample weights and $\ell_i$ is the classification loss for sample $i$. While both CL and SPL improve over the baseline SimGCD (60.1/33.2 on Real/Painting domains), achieving 62.0/34.7 and 62.8/35.0 respectively for 'All' classes, they still fall notably short of our method's performance (64.4/42.1 as shown in Table S1 and Table 6 in the main paper). In contrast, our pre-computed domain shift-based sampling achieves better results, demonstrating the effectiveness of using domain shifts as a proxy for learning difficulty.

Table S1: Evaluation on DomainNet. Bold values represent the best results.

[S8] Bengio, Yoshua, et al. "Curriculum learning." ICML. 2009.

[S9] Kumar, M., Benjamin Packer, and Daphne Koller. "Self-paced learning for latent variable models." NeurIPS. 2010.

The performance gain on DomainNet is considerably smaller than on SSB-C, which fails to highlight the advantages of the proposed approach.

We appreciate the reviewer's observation regarding the performance differences between SSB-C and DomainNet. We would like to clarify several important points:

First, DomainNet presents a highly challenging scenario due to its dramatic domain shifts and large-scale nature. We find that even ensembling of a number of SoTA domain adaptation methods, combined with SimGCD, show limited improvement on this dataset. For instance, in Table 5, SimGCD+MCC+NWD yields 35.7 ACC, compared to 42.5 for HiLo. Our method achieves 16% improvement in proportional terms for "All" classes ACC on the "Painting" domain, substantially greater than those achieved by combining two SoTA domain adaptation.

Second, the relatively smaller performance gain on DomainNet aligns with a well-known phenomenon in machine learning: improvements on large-scale datasets are typically more modest compared to smaller datasets. For instance, in object detection, SOTA improvements on COCO (large-scale) typically show 1-2% gains [S2], while improvements on smaller datasets like PASCAL VOC can reach 5-7% [S3]. Similarly, in image classification, recent architectural advances show 0.5-1% improvements on ImageNet but 2-3% gains on smaller datasets like CIFAR-100 [S4].

[S2] Sun, P., Zhang, R., Jiang, Y., Kong, T., Xu, C., Zhan, W., ... & Luo, P. “Sparse r-cnn: End-to-end object detection with learnable proposals.” CVPR. 2021.

[S3] Liu, Ze, et al. "Swin transformer: Hierarchical vision transformer using shifted windows." CVPR. 2021.

[S4] Touvron, Hugo, et al. "Training data-efficient image transformers & distillation through attention." ICML, 2021.

The method is sensitive to certain hyperparameters, like $r′$.

As demonstrated in our ablation study (Appendix M), the performance with respect to $r'$ exhibits a convex shape across different configurations, indicating a clear pattern rather than arbitrary sensitivity. Specifically, we found that $r'=0$ performs optimally for the original domain, while $r'=0.5$ works best for corrupted domains. This behavior is actually expected and interpretable: the original domain requires less distribution shifts ($r'=0$) since the data distribution is clean, while corrupted domains benefit from moderate augmentation ($r'=0.5$) to handle distribution shifts. Given our challenging setting of simultaneously handling generalized category discovery and domain adaptation, finding a single set of hyperparameters that performs optimally across all domains is inherently difficult. Nevertheless, our method maintains robust performance across domains even with sub-optimal hyperparameter choices (see Appendix M).

The method seems to be an assembly of prior works, lacking substantial novelty.

Please see the General Response for our response regarding novelty of the method.

There is no analysis of the disentangled domain and semantic features, such as distribution visualizations

We have visualized disentangled domain and semantic features projected by PCA in Figure 4a, which can effectively show disentanglement. This verifies that HiLo successfully learns domain-specific and semantic-specific features. Additionally, we have added t-SNE visualization to address the next comment below.

Same representation loss Lsrep on both domain and semantic features is confusing… It would be valuable to see t-SNE visualizations of domain features, semantic features, and their combination.

We thank the reviewer for raising this point! Our approach actually addresses a dual GCD problem that operates simultaneously across semantic and domain axes, particularly given that we work without explicit source/target domain splits in unlabeled data. Both semantic and domain spaces contain their own "seen" and "unseen" categories, making representation learning valuable for both aspects to achieve effective disentanglement.

Furthermore, since our PatchMix strategy inherently interweaves both semantic and domain information as part of the augmentation process, maintaining representation learning for both feature types becomes essential for proper disentanglement. We have added t-SNE visualizations in the appendix P that demonstrate that our approach learns distinct domain and semantic features.

Dear Reviewer 6eyX,

We are thrilled to note that your concerns have been addressed. We sincerely appreciate your dedicated time and effort in offering invaluable feedback.

Features are disentangled by assuming that features from different layers represent domain and semantic information….However, this assumption may oversimplify the complexity of feature representation in neural networks.

Our selection of high-level and low-level feature is based on previous literature and empirical evidence. First, several works [S5][S6][S7] have demonstrated the efficacy of separating domain and semantic information and leveraging mutual information to address domain adaptation challenges, showing it is not always that complex when introducing mutual information minimization between the representations. Furthermore, the selection of layers to represent domain and semantic features is not arbitrary but based on extensive empirical investigation. As illustrated in Figure 3 of our paper, we conduct a comprehensive analysis to determine the optimal layer assignments. Our results demonstrate that attaching the domain head to earlier layers yields superior performance, corroborating the hypothesis that lower-level features are more domain-oriented. Likewise, Figure 3(b) shows that fixing the domain head to the first layer and varying the 'Deep' layer for the semantic head from the last to the fourth last layer reveals that the last layer is optimal for the semantic head. These findings substantiate the importance of domain-semantic feature disentanglement and validate our design choice of utilizing lower-level features for domain-specific information and higher-level features for semantic-specific information. In Figure 4(a), we also present the visualization by first applying PCA to the domain features and semantic features obtained through $\mathcal{H}$, and then plotting the corresponding images. As can be seen, the images are naturally clustered according to their domains and semantics, demonstrating that HiLo successfully learns domain-specific and semantic-specific features from higher-level features and the shallower layers.

[S5] Zhao, Haiteng, et al. "Domain adaptation via maximizing surrogate mutual information." IJCAI. 2022.

[S6] Park, Geon Yeong, and Sang Wan Lee. "Information-theoretic regularization for multi-source domain adaptation." ICCV. 2021.

[S7] Sharma, Yash, Sana Syed, and Donald E. Brown. "Mani: Maximizing mutual information for nuclei cross-domain unsupervised segmentation." MICCAI, 2022.

When dealing with data with large domain differences, it is a challenge to determine the mixing proportion and application method accurately. If not handled properly, it may introduce too much noise or incorrect information, which may instead interfere with the learning process of the model and reduce the classification performance.

We agree that determining appropriate mixing proportions is crucial for PatchMix to be effective. To address this challenge, our method incorporates two dynamic mechanisms for controlling mixing proportions. First, we introduce $\alpha$, which weights each patch based on attention scores from $x$ and $x′$. This ensures that patches less relevant to semantic content receive lower weights in computing $\mathcal{L}^{rep}_s$ and $\mathcal{L}^{cls}_s$, effectively reducing the impact of potentially noisy or irrelevant patches. Second, $\beta$ is sampled from with Beta distribution (Zhu et al. (2023)), as a random mixing proportion for each patch $j$. This dual-control mechanism makes our mixing proportion dynamic rather than static, allowing the model to adaptively adjust the influence of different patches based on their semantic relevance while maintaining a degree of randomness through the Beta distribution. This approach helps mitigate the risk of introducing excessive noise or incorrect information during the mixing process, particularly when dealing with significant domain differences.