Deep Unsupervised Hashing via External Guidance

We sincerely appreciate your insightful comments and suggestions. To enhance readability, we will refine the description of the bidirectional contrastive loss in the revised manuscript accordingly.

Q1: How does DUH-EG scale when the external textual database is significantly larger, and what are the computational implications?

R1: Our method enhances scalability by selecting representative textual features and aggregating them into a single external feature e using Eq.(5) of the original paper. This ensures scalability when handling a significantly larger textual database.

Regarding computational cost and scalability, Table 1 and Table 2 analyze computational cost, showing that while pre-processing time increases with larger external noun databases, MAP performance remains stable, demonstrating scalability. Additionally, the MAP performance remains largely unchanged, indicating that our method exhibits good scalability. However, in the inference phase, since only the image hashing network is used to generate discriminative hash codes, the size of the textual database has no impact on inference efficiency.

---

Q2: Have you evaluated the impact of noisy or irrelevant external textual features on the hash code quality?

R2: Yes. We highlighted the positive impact of filtering redundant external textual features using the External Feature Construction (EFC) method (Section 3.2) in the supplementary material. As shown in Fig. 7, after filtering, the feature "Cat_sleep" had the highest weight. Without EFC, irrelevant features like "Round_shape," "Ovoid," and "Hair_ball" had higher weights than "Cat_sleep." The ablation study in Table 3 confirms that irrelevant features degrade the quality of the learned hash codes.

Q3: Can you provide further justification for the selection of the similarity thresholds $T_1$ and $T_2$ across different datasets? Are these hyperparameters sensitive to dataset characteristics?

R3:

1. To ensure consistency across datasets, we fixed $T_1$ at 0.90 and $T_2$ at 0.97 when comparing with other methods in Table 1 of the original paper.
2. No, they are not sensitive to datasets, so we use the same parameters in all datasets.

Q4: How does the choice of pre-trained models (e.g., using alternatives to CLIP) affect performance? Would similar improvements be expected with other multi-modal models?

R4:

1. Since our method uses Bidirectional Contrastive Learning to align an image's textual and visual representations, the quality of external knowledge introduced through the textual modality affects the hashing network’s performance. To validate this, we trained our model with textual features from CLIP (ViT-B/16) and CLIP (ViT-B/32). As shown in Table 3, the MAP performance differs significantly on CIFAR-10 and MSCOCO, confirming the impact of the pre-trained model choice.
2. Yes, similar improvements would be expected with other multi-modal models, as evidenced by the results of CLIP (ViT-B/16) and CLIP (ViT-B/32).

Table 1: The number of words and pre-processing time for different external knowledge.

Table 2: The MAP performance while using different external knowledge.

Table 3: The MAP performance while using different pre-trained models.

We sincerely appreciate your valuable feedback and constructive suggestions. Below are our point-to-point responses:

Q1: A discussion on the computational costs and scalability issues.

R1: To analyze computational costs and scalability, we evaluated our approach using external vocabulary sets of varying sizes from four sources: 1) Noun category labels from ImageNet; 2) Nouns extracted from WordNet; 3) Nouns from the ConceptNet knowledge graph; and 4) The full vocabulary from GloVe.

The experimental results are shown in Table 1, presenting the pre-processing time of our method for each external knowledge source. Since our approach extracts textual features for all words using CLIP, the pre-processing time increases as the vocabulary size grows significantly. Therefore, exploring more efficient strategies for extracting external textual features is a promising direction for our future research. Notably, after external knowledge preprocessing, our training and inference are the same as the existing methods with no additional time added.

Regarding scalability, we evaluated the model performance using four external vocabulary sources with different sizes, as shown in Table 2. Our External Feature Construction (EFC) method (in section 3.2 of the original paper) effectively mitigates redundancy in external textual features, ensuring robust performance even when different external knowledge sources are used. This highlights our method's adaptability and scalability.

Q2: Relying on external pre-trained models (e.g., CLIP) and WordNet might limit the method's applicability in scenarios where such resources are constrained.

R2: Our method leverages the pre-trained CLIP to incorporate textual features from WordNet as external knowledge for unsupervised hashing. However, we would like to clarify that our approach does not strictly rely on these specific resources. In resource-constrained scenarios, lightweight or domain-specific pre-trained multimodal models and textual knowledge bases could be used as alternatives.

Q3: Why was $T_2$ fixed at 0.97 for all datasets despite varying optimal values (Fig. 3b)?

R3: To maintain a consistent set of hyperparameters across all datasets, we fixed $T_2=0.97$ in comparison experiments, thereby reducing parameter tuning costs. If dataset-specific $T_2$ values were used, the performance would improve slightly, as indicated by the star-marked points in Fig. 3b.

Q4: What is the training/inference time of DUH-EG compared to baselines, especially with CLIP?

R4: We freeze the parameters of CLIP and train only the subsequent hashing layers. This makes our training process more efficient than updating all parameters of CLIP. For inference, our method consists of two steps: (1) feature extraction using CLIP and (2) hashing the features via the trained hashing layers. Moreover, the hashing layers consist of only two fully connected layers [$F$--$512$--$L$] with activation functions, which introduce negligible additional overhead (about 0.3M params). We provide a detailed overview of the computational costs associated with our method in Table 3, which demonstrates the efficiency of our method compared with CLIP.

Q5: Will code and pretrained models be released?

R5: Yes. We will release them after the acceptance of the paper.

Table 1: The number of words and pre-processing time for different external knowledge.

Table 2: The MAP performance using different external knowledge.

Table 3: Training/inference time, FLOPs, and params of DUH-EG and CLIP on an NVIDIA 3090 GPU.

We sincerely appreciate your valuable feedback and constructive suggestions. Below are our point-to-point responses:

Q1: More recent works could be included in the related work of deep unsupervised hashing.

R1: Thank you for your suggestion. We will update the related work section to incorporate more recent studies on deep unsupervised hashing, ensuring a more comprehensive and up-to-date literature review.

---

Q2: The paper is clear in its language, but it’s better to check some mathematical operations with different names (e.g., 'sim' and 'cos' in Eq.(3), (4), (6), (8), and Lines 256-258) which may have the same meanings and cause ambiguity.

R2: We appreciate your careful examination of our mathematical operations. To eliminate potential ambiguity, we will revise the notation in Eq.(3), (4), (6), and (8) of the original paper, as well as in Lines 256–258, ensuring consistency in mathematical operations.

---

Q3: The authors may limit the use of boldface abbreviations only for necessary cases.

R3: Thanks for your suggestion. We will refine the formatting to use boldface abbreviations only where necessary, improving readability and clarity.

---

Q4: Why does the similarity threshold $T_1$ specifically affect the Flickr25k dataset when its value is high?

R4: The similarity threshold $T_1$ plays a crucial role in selecting representative textual features with lower similarity to the cluster center. As shown in Eq.(5) of the original paper, the final fused external feature e is computed from the external textual feature set \left(\bar{\{t}}_w^{com}\right). Thus, reducing redundancy in \left(\bar{\{t}}_w^{com}\right) is essential for ensuring well-matched external textual knowledge for the original images.

If $T_1$ is set too high, some representative textual features may be filtered out during the process described in Eq.(4), weakening the discriminative power of the final fused external feature e. Since the representational capacity of e is influenced by \left(\bar{\{t}}_w^{com}\right), different datasets may respond differently to variations in $T_1$. In the case of Flickr25k, a higher $T_1$ can significantly impact model performance, likely due to the dataset’s inherent characteristics, making it more sensitive to the removal of certain textual features.

---

Q5: Why does a reduction in $T_2$ have a more significant impact on the model compared to $T_1$?

R5: Unlike $T_1$, which filters representative textual features, the primary role of the similarity threshold $T_2$ is to determine positive and negative sample pairs in Eq.(9). Fluctuations in $T_2$ directly affect the model’s ability to correctly distinguish between potential positive and negative pairs.

Even a slight reduction in $T_2$ can cause some negative sample pairs to be misclassified as positive ones. This misclassification directly affects the discrimination of the learned hash codes, ultimately leading to a decline in overall model performance.

We greatly appreciate your thoughtful feedback and recognition of our contributions. Below are our point-to-point responses:

Q1: How does the selection of external textual features affect the model's robustness across datasets?

R1: In our work, we focus on widely used benchmarks, such as CIFAR-10, NUS-WIDE, FLICKR25K, and MSCOCO, which primarily contain images of common objects and everyday scenes. To ensure generalization, we use a general vocabulary (e.g., WordNet) as our external knowledge source. Specifically, we extract textual features via CLIP and align them with visual features, promoting semantic consistency among similar images in the learned discrete representations.

However, due to the presence of synonyms, textual features from the general vocabulary often introduce redundancy. To address this, we propose an External Feature Construction (EFC) method (Section 3.2) to cluster semantically similar nouns and select representative textual features, thereby reducing redundancy caused by excessive synonyms. This process enhances the semantic richness of the aggregated external textual features obtained via weighted summation (Eq. 5) and improves their representational capacity.

As demonstrated in our ablation experiments (Table 3), the EFC method improves model performance. Furthermore, as shown in Fig. 7 (Appendix), applying EFC reduces the number of high-weighted nouns with similar meanings, validating its effectiveness in redundancy reduction.

---

Q2: Would using domain-specific external knowledge further enhance performance?

R2: Yes, using domain-specific external knowledge can further enhance performance. To evaluate this, we conducted an ablation study using six external knowledge sources (ImageNet, ConceptNet, WordNet_N, WordNet_V, GloVe, and Medical) on the MSCOCO dataset, which belongs to the natural image domain.

• ImageNet represents 21,843 noun labels from the ImageNet dataset.
• ConceptNet and WordNet_N contain noun data.
• WordNet_V consists of verb data.
• GloVe includes words spanning multiple parts of speech.
• Medical contains vocabulary from the medical domain.

Experimental results in Table 1 show that models trained with general noun vocabulary (e.g., ImageNet, ConceptNet, WordNet_N) perform better, despite differences in vocabulary size. In contrast, models trained on verb-based (WordNet_V) or mixed-POS vocabularies (GloVe) exhibit lower performance, with medical vocabulary yielding the lowest performance.

In summary, for a target dataset (e.g., MSCOCO) in the natural image domain, using domain-relevant external knowledge (e.g., general noun vocabulary) effectively enhances model performance. Thus, matching external knowledge with the target dataset’s domain improves performance.

Table 1: The ablation studies on the influence of different external knowledge on specific dataset MSCOCO.

Q3: Can this approach be extended to other modalities beyond images and text?

R3: Yes, our approach could be extended to other modalities, but certain preconditions apply. Specifically, our method aligns external textual features with visual representations via CLIP, making it well-suited for unsupervised image hashing. While extending to other modalities (e.g., audio using models like Wav2CLIP) is promising, its effectiveness depends on the availability of modality-specific external knowledge and corresponding pre-trained models.