Learning Attribute-Aware Hash Codes for Fine-Grained Image Retrieval via Query Optimization

Comment1: Figure 3 illustrates the curves at different values of c, and the authors further support their analysis with the visual results in Figure 4. However, Figure 4 only provides visualization for the specific case of C=200. Does the trend of the loss landscape at different values of c align with that in Figure 3? More visual results should be included.

Reply1: We sincerely appreciate your constructive feedback. In response to your insightful suggestion, we have supplemented the analysis with additional visualization cases across multiple c-values (C = 200, 292, 555). These extended results consistently demonstrate that:

1. The loss landscape exhibits a strong correlation with $\mu$, aligning well with the trends shown in Figure 3.
2. Larger class numbers coupled with lower feature dimensions indeed lead to:
   • Inaccessible lower local minima
   • Less smooth loss landscape

This phenomenon aligns with our ​conjecture on the limitation of large class numbers with low feature dimensions. The visualization results can be found at: https://anonymous.4open.science/r/rebuttal_for_ICML2025-89B6/figure1.png

---

Comment2: Section 3.3 mentions that without the auxiliary branch, the model may not learn discriminative features. Are there any experimental results to support this observation? The authors could consider plotting the t-SNE result of X_out to observe the feature distribution across different categories.

Reply2: We sincerely appreciate your insightful suggestion. To validate our observation, we have conducted t-SNE visualization on the feature embeddings $X_{out}$. The results demonstrate that when training ​with the auxiliary branch, the feature distributions exhibit:

1. ​Larger inter-class distances: Distinct categories form more separated clusters.
2. ​Smaller intra-class variations: Samples within the same category show tighter aggregation.

In contrast, the model ​without the auxiliary branch produces features with substantially overlapping distributions across different categories. This quantitative evidence strongly supports our claim that the auxiliary branch enhances feature discriminability. visualization results can be found at: https://anonymous.4open.science/r/rebuttal_for_ICML2025-89B6/figure2.png

---

Comment3: There is a spelling error in line 302.

Reply3: We sincerely appreciate your thorough review. We have carefully addressed the identified issues and conducted further proofreading of the manuscript to ensure its quality.

Comment1: The paper lacks a comparison with the recent method ConceptHash

Reply1:

Table 2:Comparison of Retrieval Accuracy (%mAP) with ConceptHash.$^+$ denotes our method is trained with classification and uses the ViT-Base backbone.

Due to the different experimental settings between ConceptHash and our experiment (both in the main paper and appendix). We made adjustments to our own method and re-conducted the experiment. The experimental results are shown in Table 1, demonstrating that our method is comparable to these methods.

---

Comment2: The paper discusses the limitation of large class numbers with low feature dimensions. However, the experiments related to the hyperparameter N, which are directly relevant to this analysis, were only conducted on one dataset, which is very insufficient.

Reply2:

Table 2: Comparison results of hyperparameter N. Results are based on five commonly used benchmark datasets under the 12-bits setting.

Thank you for your suggestion. We have provided more experimental results in Table 2. The results on different datasets show that as $N$ increases from 1 to 6, the retrieval performance improves rapidly. Once $N$ exceeds 6, the performance stabilizes, which follows the same trend as the change of mu shown in Figure 3.

---

Comment3: The authors provide a detailed description of the motivation behind the auxiliary branch design, but only a brief description is given regarding the specific operations.

Reply3: For the specific implementation of the auxiliary branch, we provide a detailed description. Given a query $q_i \in \mathbb{R}^d$, we divide $q_i$ evenly into multiple parts. We then perform a circular shift with step sizes of $1, 2, 3, ..., N-1$. For each of the $N-1$ different step sizes, we extend the original $q_i$ into $N-1$ new queries, $\hat{q}_i^j$, where $j = 1, 2, 3, ..., N-1$ indexes the different step sizes and all queries share the same parameters. We refer to this extension process as query transformation. We initialize a total of $k$ learnable queries. After the query transformation operation, each query is passed to the decoder for computation according to Equation 3. Finally, we obtain $\hat{h} \in \mathbb{R}^{N \times k}$, which is optimized using the loss function defined in Equation 6.

Comment4: The Related Work section on 'Set Prediction and Parallel Decoding' could be divided into two parts.

Reply4: We sincerely appreciate your valuable suggestion regarding the organization of the related work section. We have restructured this part to improve clarity.

---

Questions1: What is the difference between Query Optimization mentioned in the paper and c-vectors optimization in CMBH？

Reply1: Our approach differs from CMBH in the following ways:

1. CMBH’s contribution focused on better extracting subtle image features but does not improve the process of generating hash codes. This distinction results in the hash codes generated by CMBH not exhibiting interpretability, while our approach generates hash codes that can indicate whether an image possesses a certain visual attribute.

2. We primarily follow the pairwise setting, which means that we can not use the classification task for training. In contrast, CMBH’s optimization process is designed based on a classification task, which cannot be used in the pairwise setting.

Comment1: The subtle feature extractor employs a multi-scale framework in conjunction with a multi-head self-attention (MHSA) mechanism. This method is widely adopted in deep learning, particularly for image feature extraction. For instance, architectures like Mobile-Former and ConVit exemplify this trend. While the design is reasonable, it does not offer significant innovation.

Reply1: We would like to emphasize that:

1. We model the hash problem as a set prediction problem, where each element in the set represents a bit of the hash code that can indicate a visual attribute. Specifically, our method uses $k$ learnable queries to directly decouple distinguishable visual attributes from complex feature representations to generate the $k$-bit hash code.

2. Previous fine-grained retrieval methods have overlooked the low-bit problem. We provide an analysis from the perspective of cosine similarity and design a query transformation strategy that effectively alleviates this issue.    

3. Regarding the design of the subtle feature extractor, our main objective is to use a lightweight and simple strategy to extract fine-grained features. Overly complex feature extraction modules also lead to larger model parameters and greater computational overhead. Some previous methods have primarily focused on the design of this module, while neglecting improvements in the hash code generation process.

Overall, the first two key contributions have never been explored in previous work. At the same time, using our subtle feature extractor makes the method more lightweight and reduces computational overhead..

---

Comment2: Line 58 should use 'demonstrates,' and line 75 should use 'are'.

Reply2: We sincerely appreciate your thorough review. We have carefully addressed the identified issues and conducted further proofreading of the manuscript to ensure its quality.

---

Questions1: How many layers of cross attention does the decoder part consist of?

Reply1: The decoder consists of a single layer. Its lightweight design ensures that the introduction of auxiliary branches does not incur significant computational overhead.

Comment1: One of the main contributions of the paper is the introduction of an additional auxiliary branch. However, the description of this operation is too brief and does not sufficiently clarify its specific implementation. It is recommended that the authors provide a more detailed explanation of the auxiliary branch, including the interaction between queries and features, so that readers can better understand the significance and practical implications of this contribution.

Reply1: We sincerely appreciate your valuable feedback. Without the auxiliary branches, the learnable queries interact with the extracted features in a fixed manner. For instance, each query can be split into $N$ segments. When $N=2$, the first segment consistently interacts with features corresponding to the first half of the image channels, while the second segment interacts with features from the latter half. However, when an auxiliary branch is introduced, the interaction pattern undergoes a significant change. In the case where $N=2$, the first segment of the query can interact with features from either the first half or the latter half of the image channels, and the same applies to the second segment. The introduction of auxiliary branches expands the receptive range of queries across the channel dimension. Since different channels typically encode distinct semantic information or visual features, this enhancement allows the optimized queries to better capture visual attributes across different images.

---

Comment2: Figure 7 could provide a more detailed description.

Reply2: We sincerely appreciate your valuable feedback. In Figure 7, we present the visualization results of heatmaps across different datasets. Each row corresponds to a distinct learnable attribute query ($q_i \in \mathbb{R}^d$). The heatmaps generated by different $q_i$ demonstrate that the learned queries effectively focus on different parts of the objects. For instance, on $CUB200$ dataset, certain queries attend to the body, while others focus on the beak of a bird. On $Aircraft$ dataset, some queries focus on the wings, while others focus on the tail of the airplane. These visualizations qualitatively illustrate that our proposed query learning achieves strong interpretability.

---

Questions1: What is the relationship between the two decoders in Figure 4?

Reply1: Figure 4 shows the visualized results of the loss landscape under different conditions. In Figure 5, the two decoders share the same weights.