Prototypical Hash Encoding for On-the-Fly Fine-Grained Category Discovery

We sincerely thank the reviewer’s recognition of our method’s good performance and its robustness to varying coding lengths (sensitivity issue). We value the reviewer’s insightful comments and will incorporate these into our final revision.

Q1: Writing. Thanks for your constructive suggestions. We will leave the section of related works to the main text, and carefully check and improve the expressions in the revision.

Q2: Technical contribution.

We argue that our contribution mainly lies in identifying and effectively mitigating the high sensitivity issue existing in current hash-based OCD methods, which was acknowledged by Reviewers 6xA4 and Pa9u as well. We provide elaborated discussions as follows.

1. Compared to prototype learning method (SimGCD [1]), our PHE has two main advantages. First, unlike SimGCD that learns only one prototype for one category, our PHE generates multiple prototypes for one class, which is favorable for modeling intra-class variance of fine-grained categories. Empirically, we have supplemented experiments with the prototype learning method used in SimGCD to validate the superiority of our methods as shown in the table below.

Specifically, due to lack of unlabeled data, we removed the $L_{cls}^u$ and the prototypes corresponding to new categories in SimGCD. In the table, "SimGCD+PHE" indicates that we used SimGCD for prototype learning while also mapping the prototypes learned by SimGCD to hash centers for category encoding. This approach yielded very poor results on both datasets. "SimGCD-MC+PHE" refers to the use of manually obtained centers that satisfy the Gilbert-Varshamov bound and features from the SimGCD projection head for category encoding. Compared to mapping SimGCD’s prototypes to hash centers, the SimGCD-MC+PHE variant shows an average improvement of 8.8% on two datasets, demonstrating that the prototypes learned by SimGCD are not suitable for category encoding in fine-grained OCD scenarios. Second, Unlike the prototypes in SimGCD which are weights of classifiers, the learned prototype in PHE can be explicitly visualized, which provides additional perspective for interpreting the model's behavior, as illustrated in Fig.3 of the main text.

2. Compared with deep hash method (MDSH[2]), our PHE has two main differences: First, unlike MDSH that pre-calculates hash centers and fixes these centers during training, our hash centers are derived by mapping from category prototypes and then are updated by end-to-end optimization, which better preserves the relationships between fine-grained categories learned in the prototype feature space. Second, we design an additional Hamming ball-based inference tailor-made for OCD, which effectively mitigates the sensitivity issues associated with using hash codes. Furthermore, we have supplemented experiments and the results in the following table verify the superiority of our PHE. We also provide the results of more hash-based baselines in Q1 of Reviewer 6xA4.

Q3: Motivation of instance discrimination and class discrimination

We agree that both OCD and GCD [3] need to learn good representations with “optimal balance between instance discrimination and class discrimination”. However, our primary motivation comes from the identification of the high sensitivity issue caused by applying hash codes in OCD tasks, which significantly hinders the effectiveness of the current hash-based OCD method. Motivated by this, we present the specific prototype design, loss implementation and Hamming-ball-based inference, to mitigate the high-sensitivity issue.

Q4: Technical practice of prototype-based visualization examples. The process mainly consists of three steps: 1) Input an image to obtain its feature representation, while also capturing the attention map during forward propagation. 2) Select the top-k most similar (activated) prototypes from all prototypes. 3) Visualize the original samples/images corresponding to these prototypes. We will include detailed examples and process descriptions in the revision.

Q5: Will the semi-supervised contrastive loss (SSCL) perform? Firstly, we'd like to clarify that in the OCD setting, only labeled data of known classes is available for model training, and unlabeled data appears only in an on-the-fly format during testing. Thus, the SSCL used in GCD[1] is incompatible with OCD. As for the experiments in Tab. 3, in fact, we did not include experiments with "supcon and unsupcon baselines". “Supcon Cls” represents the use of classification methods based on supervised contrastive learning for representation learning. According to the results, we find that although this variant performs well on seen categories, their generalization capabilities are inferior to our full prototype-based method.

[1] Parametric Classification for Generalized Category Discovery: A Baseline Study. ICCV 2023

[2] Deep Hashing with Minimal-Distance-Separated Hash Centers. CVPR 2023.

[3] Generalized Category Discovery. CVPR 2022.

We sincerely thank the reviewer’s recognition of our motivation to address the limitations of previous OCD models. We value the reviewer’s insightful comments and will incorporate these into our final revision.

Q1: Sharing the same core idea of utilizing hash codes for OCD. We indeed use hash codes as category descriptors, which are also widely used in image retrieval and deep hashing methods. However, we argue that our innovation lies in: 1. Identifying the high sensitivity issue when applying hash codes to the challenging fine-grained OCD task, where only data from known categories is available. 2. We have introduced a new OCD framework, PHE, that explicitly achieves inter-class separability and intra-class compactness, compared to the SOTA method SMILE. 3. Unlike SMILE, we designed an on-the-fly inference method based on the Hamming ball. Our PHE method effectively mitigates the high sensitivity problem, for example, surpassing SMILE by 15.5% in terms of all accuracy on the CUB dataset with hash code bits = 64. This point has also been acknowledged by Reviewers 6xA4 and Pa9u.

Q2: The reasons why prototypes can address the sensitivity issue. Thanks for your insightful comments. We'd like to explain as follows.

1. In fact, instead of depending on only prototypes themselves, we mitigate the sensitivity issue by 1) constraining the hash centers (mapped from feature-level prototypes) to be at least a Hamming distance of $d_{max}$ apart based on the Gilbert-Varshamov bound; 2) representing a category using a Hamming ball of radius $max(\lfloor \frac{d_{max}}{2} \rfloor, 1)$.

2. The role of prototypes is to: 1) achieve better representation learning; 2) map the category prototypes to category hash centers, unifying representation learning with hash encoding. The importance of prototype learning is evident from the variant without prototype learning loss $L_f$ in Tab.2 of the main paper. Removing the prototype learning loss $L_f$ results in an obvious accuracy drop on datasets in terms of all, old and new accuracy.

We will clarify this point more clearly in the introduction.

Q3: Feature-level prototypes vs. Hash code-based prototypes

1. Experimental results with a powerful feature-level prototype learning method. Firstly, we would like to further explain that in the OCD setting, test data appear one-by-one, and the OCD task requires the model to instantly assign a category to the test data. Therefore, if only feature-level prototype learning is used, the method needs an additional online clustering approach to yield real-time category descriptors, which is also acknowledged in the original OCD paper [1]. Thus, we adapt SimGCD [2] ( an advanced prototype learning approach for GCD, which was also acknowledged by Reviewer dwc8) into OCD settings and implement two variant approaches: 1) SimGCD attached a online clustering approach SLC in [1] to achieve instance-wise inference. 2) SimGCD with our hash prototype framework. The experimental results below show that introducing PHE significantly improved all accuracy across two datasets. Meanwhile, our full method achieves better generalization ability, especially on new categories.

2. Advantages of hash code-based features. Hash code-based features can directly use the features’ signs (hash codes) as category descriptors, where features with the same category descriptor are considered to be of the same category. Therefore, compared to feature-level prototype learning, introducing a hash code-based design can directly optimize category descriptors, achieving higher accuracy in the OCD setting. According to the results in Table 1 of the main paper, where only feature-level prototype learning results (SLC) and hash code-based prototypes learning (our PHE) are compared, introducing hash code-based prototypes improved the average accuracy by 8.2% across eight datasets.

Q4: Training efficiency. We provided a comparison of training times between our PHE and the SOTA method, SMILE, measured in minutes, as shown in the table below. To ensure fairness, all experiments were conducted on an NVIDIA RTX A6000 GPU. Both algorithms were trained for 200 epochs using mixed precision training. The dataloader parameters were consistent, with a batch size of 128 and num_workers set to 8. According to the table, our average training time across four datasets is less by 45.8 minutes compared to SMILE. This is primarily due to SMILE’s use of supervised contrastive learning with two views of samples for representation learning, which requires higher computational resources.

Q5: Hyperparameter section. To avoid over-hyperparameter tuning, we use a fixed set of hyperparameters which are obtained based on CUB to report accuracy for all datasets, without tuning them individually for each dataset.

If we have misunderstood any of your concerns, please let us know in the future comments.

[1] On-the-fly category discovery. CVPR 2023.

[2] Parametric Classification for Generalized Category Discovery: A Baseline Study. ICCV 2023

We sincerely thank the reviewer’s recognition of our motivation and experimental results in addressing the hash sensitivity issue. We value the reviewer’s insightful comments and will include these into our final revision.

Q1: Improving accuracy in recognizing unknown categories.

1. Limited room for improvement due to the unique challenge of OCD. Compared with GCD/NCD tasks, OCD tasks do not use unlabeled data that might include unknown-category data for model training, which leads to a larger challenge for OCD methods to recognize unknown categories. We discussed the limitation in the main paper. Compared to the SOTA method, SMILE, our PHE achieves better accuracy for unknown categories. On the CUB dataset, our method exceeds SMILE by 4.1%/16% on 12/16 bits, respectively. Thus, as recognized by Reviewer ZQiT and dwc8, such an improvement might be non-trivial for OCD.

2. A possible solution in future work. Due to the constraint of training data in OCD setting, we consider introducing additional knowledge from pre-trained Large Language Models (LLMs). Firstly, we can leverage LLMs to establish a bank of category attribute prototypes that are expected to be shared across both known and unknown categories. Then, during the on-the-fly prediction process, we plan to use LLM+VLM to match the attribute prototypes for unknown categories. Finally, by jointly considering the instance and attribute features, we hope that our PHE can generate more accurate predictions.

Q2: Computational cost for large-scale datasets. We agree that the computational cost of PHE is relatively high; however, the computational cost mainly depends on the number of known categories, which is not directly related to dataset scale. Specifically, each known category has a hash center, thus hash centers are represented by a tensor $\mathbf h$ with a shape [num_class, bit]. Main calculations involve simple two-dimensional matrix multiplications, which include the dot product of features $\mathbf{b}$ with a shape [batch_size, bit] and hash centers, $\mathbf{b} * \mathbf{h}$, as well as the dot product operations in the calculation of Hamming distances $\mathbf{h} * \mathbf{h}^\text{T}$. Empirically, we have verified the analyses in the table below. It shows that although the Food dataset is larger in scale than CUB-200 and Scars-198, it contains only 100 categories. Therefore, the average training time per sample for the Food dataset is significantly lower than that for CUB and Scars.

Q3: The impact of different K values on the model. We regard the "K categories" in the review comment as the number of prototypes per category, $k$. We have detailed the impact of this hyperparameter in Fig. 4 and Sec.3.4 of the main paper. Specifically, $k=1$ results in suboptimal performance, as it doesn't effectively represent the complexity of a category. Using larger $k$ captures the nuances within a category, which is crucial for fine-grained categories. However, when $k>5$, the improvement is minimal, as validated on two datasets. Therefore, K is not sensitive, and we choose a relatively large value of 10 for all datasets to avoid over-tuning this hyperparameter.

Q4: The impact of $\theta$ in the masking strategy. We have added experiments on the two datasets and reported results in the table below. The masking strategy helps reduce redundancy when using multiple prototypes, thus facilitating prototype learning. A smaller $\theta$ value is found to be appropriate; within the range of (0, 0.2), the masking strategy can improve accuracy. When $\theta$ exceeds 0.2, suboptimal results occur on both datasets. We did not fine-tune this parameter for each dataset but instead set it at 0.1 across all datasets.

Q5: Results of different dataset splits. Good suggestion! We added experiments using different proportions of old category selection on the CUB and SCars datasets below, with all accuracy reported and code bits=12. Based on the results, when the proportion of selected old categories is 75%/25%, our PHE outperforms SMILE by an average of 5.95%/1.0%. This indicates that our PHE is more capable of modeling the nuanced inter-category relationships in fine-grained OCD, as the number of categories increases. Results across all datasets and more code bits will be supplemented in the revision.

Q6: Hyperparameter section. 1) To avoid over-tuning hyperparameters, we use a fixed set of hyperparameters for all datasets (obtained based on CUB) to report accuracy, without tuning them individually for each dataset. 2) The hyperparameters we use are relatively robust within a certain range. In the loss function, the difference in loss scale determines the scale of the proportions.

Authors have partly addressed my concerns regarding model's stability and split variation, I'm willing to raise my score by one.

Dear Reviewer Pa9u,

We greatly appreciate your satisfaction with our responses! We will include the important discussions mentioned above in the final manuscript and highlight them. We truly appreciate your efforts!

We sincerely thank the reviewer for acknowledging that our idea is clever and simple, and for the positive comments on the writing and experiments.

Q1: More baseline with deep hash methods.

Following the meaningful suggestion, we have conducted additional comparative experiments with various deep hashing methods on CUB and SCars datasets. The results are reported and summarized as follows.

*"MC" represents manually obtained centers that meet the GV bound. "-" means results were unavailable due to the requirements of the Hadamard matrix.

Summarization of Experimental Results.

1. Comparison with methods[1-3] that generate suboptimal hash centers for modeling category information. Unlike these methods[1-3], which are designed for retrieval tasks and mainly focus on instance-level discrimination, our PHE leverages the GV bound to constrain the learning of hash centers for category discovery. This approach allows the models to be flexible enough to preserve the rich category information contained in feature-level prototypes, while also generating discriminable hash category discriminators. As a result, PHE achieves better results.
2. Comparison with [4] that optimizes only hash code to align with pre-calculated hash centers. Different from [4] that uses fixed hash centers, our hash centers are mapped from the prototypes of each category and then updated by end-to-end optimization, which can alter the relationships between categories learned in the CPG module, yielding better results compared to MDSH. For example, with code bits=12, our PHE surpasses MDSH by an average of 2.3% across two datasets. Besides the difference in hash center design, our Hamming-ball-based inference design effectively mitigates the hash sensitivity issue. For instance, when code bits=16, the MDSH-Hamming ball exceeds MDSH by an average of 2.75% across two datasets.

Q2: Comparison with OrthoHash [2] in terms of loss function.

We agree with that both PHE and OrthoHash utilize cosine similarity to constrain hash features, yet there are significant differences in our design motivations and specific implementations:

1. Differences in design motivation. OrthoHash focuses on instance retrieval and applies constraints in only the low-dimensional hash space, which tends to lose category-specific information (CSI), which is especially crucial for OCD tasks. In contrast, our hash feature constraint loss $L_f$ and prototype learning loss $L_p$ are applied in hash space and feature space, respectively. The motivation is to capture CSI as rich as possible in high-dimensional feature space. We empirically find that this design can effectively mitigate CSI loss when projecting to hash space.

2. Differences in implementation. a) Hash center generation. Our hash centers are mapped from category prototypes, which preserves the relationships between categories learned in the prototype feature space. OrthoHash uses a matrix whose row vectors are mutually orthogonal as hash centers, which might hinder modeling the relationship between similar categories in OCD tasks. b) Cosine similarity. We compute cosine similarity between hash features and hash centers without sign quantization. In contrast, OrthoHash calculates cosine similarity between continuous hash features and binary hash centers. c) Update of hash center. Our centers can be optimized during the training process, while the centers in OrthoHash are fixed.

Our PHE effectively unifies representation learning and category encoding, achieving better OCD accuracy than OrthoHash. As shown in the table of Q1, PHE surpasses OrthoHash by an average of 9% when code bits=12.

[1] Deep Polarized Network for Supervised Learning of Accurate Binary Hashing Codes. [2] One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objective. [3] Central Similarity Quantization for Efficient Image and Video Retrieval. [4] Deep Hashing with Minimal-Distance-Separated Hash Centers.