ConceptHash: Interpretable Fine-Grained Hashing with Concept Discovery

1. The title of the paper might be somewhat confusing (the title in OpenReview and the paper is different). After reading the paper, it becomes apparent that the proposed method generates hash codes for fine-grained image retrieval, which aligns more with "retrieval" than "generation." Therefore, the title could be revised to better reflect its purpose and avoid misleading readers into thinking it's a generation task.

2. While interpretability of hash codes is crucial for fine-grained retrieval, the proposed method may not be considered highly novel. For instance, a previous work [1] also introduced concept-level interpretability for hash codes, although they referred to it as "instance-level" in their paper. The primary distinction between the proposed method and [1] lies in ConceptHash's utilization of CLIP to obtain semantic information, as opposed to explicitly generating object proposals.

[1] Instance-aware hashing for multi-label image retrieval. TIP, 2016.

3. The utilization of pretrained vision-language models, such as CLIP, for enhancing image retrieval is a common approach currently. The authors assert that “…CLIP … ensures that our learned hash codes are not only discriminative within fine-grained object classes but also semantically coherent.” However, CLIP is pretrained on image-text pairs, and the model processes image/text as a whole during learning, potentially emphasizing coarse-grained knowledge. Therefore, it would be beneficial for the paper to clarify how the proposed method addresses the challenges of fine-grained hashing and specify its uniqueness compared to the existing utilization of CLIP in image retrieval.

The automatic discovery of human-understandable concepts (sub-codes) is a critical aspect of ConceptHash. However, it is an implicit learning process, the robustness of this process remains an open question. How sensitive is it to variations in training data or domain-specific nuances?

The motivation of introducing learnable concept tokens with image patch tokens seems rational. However, it lacks an explicit experimental validation to substantiate the initial premise.

Certain aspects of the paper lack rigor, such as the role for a learnable $\hat{W}_{c}$ in the fourth loss function and the interpretation of the ablation study pertaining to it.

Fairness of comparative experiments：1）Contrasting with methodologies like SEMICON and A2-Net that employ a CNN as the backbone, this paper leverages a pre-trained CLIP model. A noteworthy divergence in performance is observed between the two approaches. 2）Given the extensive training data of CLIP, which likely encompasses fine-grained data such as CUB200, the effectiveness of the idea proposed in this paper warrants further validation, such as through the substitution of different backbone architectures.

1. While the method provides interpretable tokens, the human interpretability is given by the visualization of relevant image crops (Fig. 3). This is not a problem per-se but becomes such when the number of concept tokens may lead to overfitting, thus making the tokens not interpretable. The manuscript fairly points to this weakness in Section 5 and Fig. 9.
2. Following on the previous, analyses on interpretability are held only qualitatively. To strengthen the claim of better interpretability, it would have been helpful to conduct quantitative studies, either as user studies (e.g. whether two images are of the same category based on the highlighted regions) or using explainability metrics (e.g. remove and classify) and/or using part annotations available (e.g. in CUB) to verify if parts of the birds are localizable via the learned concepts. At the moment, the reader has to assess the interpretability via the few available examples and, while they are indeed interesting and explicit, quantitative analyses would confirm the consistency in achieving better interpretability.
3. From the article, I did not find clear how the hash function $h$ is implemented. For clarity, it would be helpful to expand on it (e.g. after Eq. (3)).
4. There is no ablation on the use of adapters (Appendix A). While they are not the main contribution of the paper, it would have been helpful to show if the performance is affected by the particular choice of adapters for both the presented method and the baselines (as fine-tuning the backbones may lead to overfitting and worsen their results). Similarly, the impact of the specificity embeddings is unclear as not quantitatively ablated at the moment.
5. In principle $L_{cls}$, $L_{quan}$, and $L_{cd}$ are "redundant" as they all provide semantic supervision, even if at different levels. Table 3 at the moments shows how $L_{quan}$ and $L_{cd}$ interact, but it does not show the impact of $L_{cls}$ and its interaction with the other losses (e.g. considering two of them at the time). It would be helpful to expand the table to provide the full picture.
6. From the results/benchmarks, it is not clear if ConceptHash generalizes to classes not present during training. While it would be interesting to provide quantitative analyses on this, even a discussion on potential generalization issues would help in expanding the possible directions for future work and provide insights on the model.

Minor:

• $L_{cs}$ vs $L_{cd}$ in Tab. 3.
• Red font at the end of Section 1.

1. In Fig.3, it shows that different concept tokens clearly focus on different visual concepts. However, the meaning of each sub-code is unclear for me. For example, I can’t distinguish the difference of (00,01,10,11) in Fig.3 (a). Would more bits for each concept help to distinguish the sub-code?
2. In Fig.4, I found that ConceptHash well attention to the object, while the attention region seems not discriminative.

Others:

1. The title is “Interpretable hashing for fine-grained retrieval and generation”, but there is no “generation” content in this paper.
2. In Eq.3, should the dimension of E_m be (1, D) ?
3. In Eq.7, b_n  b_i ?
4. Typo: Sec. 4.2, an extra 'A' in line 3.