QuRe: Query-Relevant Retrieval through Hard Negative Sampling in Composed Image Retrieval

We sincerely thank Reviewer c7Tq for the positive feedback and for recognizing the novelty of our work in both the proposed method and the dataset.

Q1 : There may be cases where there are no hard-negative samples?

It is true that the number of hard negatives can vary significantly depending on the query (e.g., if the user asks for a "blue short-sleeve t-shirt," then "blue long-sleeve shirts" might be considered hard negatives) and the available candidate images in the corpus (e.g., if there are no blue shirts, no hard negatives might exist). However, identifying true hard negatives for all queries is impractical in both real-world applications and existing CIR datasets. Instead, we empirically demonstrate that QuRe improves model performance by selecting relatively hard negatives from the corpus.

Q2 : Ablation studies for the BLIP-2 model architecture

We conducted additional experiments using both BLIP and BLIP-2 architectures to ensure a fair comparison between QuRe and other baselines. Specifically, we trained QuRe on the BLIP backbone to compare with Bi-BLIP4CIR. For CoVR, we identified its latest version, CoVR-2[r1], which adopts BLIP-2 as its backbone, and compared it with our original QuRe model. Notably, the current state-of-the-art method SPRC originally used BLIP-2, and QuRe already outperformed it in our original submission. The results are presented across the following datasets:

• CIRR: QuRe outperforms Bi-BLIP4CIR and CoVR-2 with average Recall improvements of 11.53 and 6.04, respectively.
• FashionIQ: QuRe outperforms Bi-BLIP4CIR and CoVR-2 with average Recall improvements of 4.50 and 3.41, respectively.
• CIRCO: QuRe outperforms Bi-BLIP4CIR and CoVR-2 with average mAP improvements of 17.12 and 0.53, respectively.
• HP-FashionIQ: QuRe outperforms Bi-BLIP4CIR and CoVR-2 with higher human preference rates of 7.95 and 2.56, respectively.

Please refer to our response in $\text{\color{blue}**Q2 under Reviewer 6GLy**}$ for detailed results.

Q3 : Report on the improvement of false negative results

We agree that providing additional qualitative results, similar to $\text{\color{red}**Figure 6**}$, helps illustrate the improvements in handling false negatives. Using the given query, we extracted the top 4 retrieved results from Bi-BLIP4CIR, CoVR-BLIP, SPRC, and our method, QuRe. The results are shown in the following figure:

https://anonymous.4open.science/r/QuRe-ICML-Rebuttal/QuRe_FalseNegatives.png

As shown, QuRe successfully retrieves relevant t-shirts that match the query attributes, such as navy blue color (close to black, like the query image) and the presence of a chicken or bird in the center. In contrast, other baselines retrieve less relevant images, such as those in light blue or without the chicken/bird. Such irrelevant results can lead to user dissatisfaction, as user satisfaction is generally proportional to the number of relevant items in the retrieved set. These results highlight the effectiveness of QuRe in assigning higher scores to relevant images by explicitly addressing false and easy negatives during training.

Q4 : Add target image and novelty of the method in Figure 1

Thank you for pointing this out. We agree that highlighting the target image for each query in $\text{\color{red}**Figure 1**}$ would improve clarity and overall understanding. Additionally, we will include the $\text{\color{blue}**figure provided in Q3**}$ (QuRe_FalseNegatives.png) in the appendix to illustrate the effectiveness of our method, QuRe, compared with existing baselines.

Q5 : How is sharpness defined, and what if multiple sharp drops exist?

To determine whether the drop is sharp, we compute the relevance scores of all candidate images for each query. We then sort the scores and identify the top-2 largest drops occurring after the target. The corresponding images are selected as the hard negative set. An example of these steep degradation patterns is shown in $\text{\color{blue}**Q2 under Reviewer 13Df**}$ (QuRe_steep_one.png). While there may be multiple sharp drops, we believe that the top-2 degradations after the target likely represent the boundaries between false negatives, hard negatives, and easy negatives. We agree that more advanced approaches to identifying such transitions could be explored in the future. We will include this as a discussion point in the final version of the paper.

We will update the manuscript to incorporate the feedback discussed in this rebuttal. We sincerely hope that our responses, along with the originality of our contributions, have addressed your concerns. If any questions remain, we would be glad to offer further clarification.

We sincerely thank Reviewer 6GLy for recognizing the contributions of our work and for providing valuable suggestions.

Q1 : Mathematical justification for the Hard Negative Sampling strategy

We re-emphasize an important limitation in CIR datasets. Since only one or a few target images are annotated per query, it is not feasible to accurately determine which images are false, hard, or easy negatives. Hence, we first provide a theoretical justification for our approach, which is why defining the hard negative set and sampling from it in each epoch is effective in our scenario. The justification shows sampling negative from the hard negative set yields a higher expected loss than sampling from all corpus, thereby guiding the model to focus more on the defined hard negatives during training. Please refer to the following link:

https://anonymous.4open.science/r/QuRe-ICML-Rebuttal/QuRe_Justification.pdf

However, if the defined hard negative set is improperly constructed and includes false negatives, it can misguide training. In $\text{\color{red}Figure 4}$, we observe a clear performance improvement immediately after introducing the hard negative set, compared to sampling from the full corpus. Conversely, sampling from the Top-K set, which contains more false negatives than our defined set, leads to performance degradation.

Q2 : Ablation studies for the BLIP-2 model architecture

We conducted additional experiments using both BLIP and BLIP-2 model structures. Specifically, we trained QuRe with the BLIP backbone to compare with Bi-BLIP4CIR. We also identified the latest version of CoVR-BLIP, CoVR-2[r1], which uses BLIP-2, and compared it with our original QuRe model. We report results on the CIRR, FashionIQ, HP-FashionIQ, and CIRCO datasets. Notably, the current state-of-the-art method SPRC originally used BLIP-2, and QuRe already outperformed it in our original submission.

The results show that QuRe with a BLIP backbone consistently outperforms Bi-BLIP4CIR. CoVR-2, which uses a BLIP-2 backbone, still underperforms our original QuRe with BLIP-2.

[r1] Ventura, Lucas, et al. "CoVR-2: Automatic Data Construction for Composed Video Retrieval." IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).

We will revise the manuscript to incorporate the points addressed in this rebuttal. We hope our responses and the novelty of our contributions have sufficiently addressed your concerns. If there are any remaining issues or points that need further clarification, we would be more than happy to provide additional details.

We sincerely thank Reviewer 13Df for the insightful comments and the time dedicated to reviewing our manuscript. Below, we provide detailed responses to each of your points.

Q1 : Justification and thorough study to the motivation of the proposed method

The primary motivation behind our method stems from a key limitation of existing CIR datasets, where only one or a few target images are annotated per query. As a result, false negatives are often included as negatives during training. Existing baselines typically treat all non-target images as negatives, which delays convergence and degrades performance [r1].

Our method addresses this issue by modifying the contrastive learning objective to compare the true positive against a single negative with increasing difficulty. This is achieved by selecting a small set of hard negatives for each query. We validate the effectiveness of our approach by achieving state-of-the-art performance on both CIRR and FashionIQ, and demonstrating the highest correlation with human preferences on HP-FashionIQ.

We clarify the motivation of our study, we added visualizations of the steep drops in relevance scores (see $\text{\color{blue}**Q2**}$) and provided a mathematical justification in response to $\text{\color{blue}**Q1 in Reviewer 6GLy**}$.

[r1] Huynh, Tri, et al. "Boosting contrastive self-supervised learning with false negative cancellation." Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2022.

Q2 : Visualizations and discussions on the steep

Thank you for pointing this out. We agree that visualizing the steeps during optimization is beneficial. To illustrate this, we plotted the relevance scores for one sample from the FashionIQ dataset at two stages: before training and after the warm-up training. Please refer to the following link:

https://anonymous.4open.science/r/QuRe-ICML-Rebuttal/QuRe_steep_one.png

We defined the hard negatives as the images between the red and green lines. By selecting the top-2 largest drops in relevance score following the target, we observed steep degradations immediately before the red line and after the green line. These steep drops indicate substantial decreases in relevance[r2], suggesting that the selected boundaries effectively separate false negatives, hard negatives, and easy negatives.

As the above figure shows only a single query, we also aggregated the results across all queries, which can be found here:

https://anonymous.4open.science/r/QuRe-ICML-Rebuttal/QuRe_steep_agg.png

This aggregated visualization demonstrates that after the warm-up phase, the hard negatives shift toward higher ranks, likely capturing more true hard negatives. This supports our design choice of including a warm-up stage before defining hard negatives. Without a warm-up, the hard negative set would contain many easy negatives, as shown in the left figure.

[r2] Xia, Peng, et al. "Mmed-rag: Versatile multimodal rag system for medical vision language models." ICLR 2025.

Q3 : Apply QuRe on a wide range of the datasets beyond the HP-FashionIQ and CIRR

We evaluated our method on four datasets: CIRR, FashionIQ, HP-FashionIQ, and CIRCO. CIRR and FashionIQ are two representative datasets in CIR, where CIRR covers general domains (e.g., people, animals, food) and FashionIQ focuses on fashion-related queries (e.g., shirt, dress, etc). We agree that evaluating on a broader range of datasets would further validate our approach. To this end, we created the HP-FashionIQ dataset to better capture human preferences. We additionally evaluated on CIRCO, a dataset derived from COCO, which enables relevance-based retrieval evaluation using mAP in a zero-shot setting.

Q4 : How to make sure the quality of the ranking and steeps especially at the initial stage?

If the model is not sufficiently trained and the relevance scores are not reliable, our proposed method, which defines the hard negative set based on relevance scores, might produce inaccurate results. To address this, we include a warm-up phase where the model is trained by sampling negatives from the entire corpus without defining a hard negative set. As shown in $\text{\color{red}**Figure 4**}$, the model achieves comparable performance even when negatives are sampled from the full corpus ("All corpus"). Additionally, the QuRe_steep_agg.png visualization in $\text{\color{blue}**Q2**}$ demonstrates that defining the hard negative set without warm-up training tends to select easy negatives, which undermines the effectiveness of hard negative mining.

We will revise the manuscript to reflect the points raised in this rebuttal. We hope our responses and the novelty of our contributions have adequately addressed your concerns. Should any issues remain, we are happy to provide further clarification.