LRVS-Fashion: Extending Visual Search with Referring Instructions

We thank the reviewer for their feedback and for recognizing the value of our task in the e-commerce domain. We address their concerns below.

W1.1 The significance of the proposed task needs further clarification. What distinguishes Referred Visual Search from text-guided visual localization or composed image retrieval?

Text-Guided Visual Localization is a localization task aimed at explicitly locating an object in an image based on textual instructions; it is not a retrieval task.

Composed Image Retrieval (CIR) is a specific case of Conditional Image Similarity (CIS), where the conditioning text provides editing instructions to retrieve a different item from a gallery, as explained in [1].

Referred Visual Search (RVS) is also a CIS task, but its goal is to extract the embedding of an existing item from a complex input image. It is similar to the unnamed “Focus Object” task in GeneCIS [1], but our benchmark is significantly more demanding, with a gallery size of over 2M images compared to only 15 in GeneCIS.

We have revised our work to better clarify this distinction between our task, CIR, and CIS.

W1.2 Can existing methods in these areas be adapted to this task?

Adapting existing CIR methods is an interesting direction. We used some of these methods as baselines and added additional ones in our revised version, none of them being competitive with our proposed conditional-ViT. See Table 2 of the revised version. Our results demonstrate that dedicated research efforts are needed to bridge the gap between these two tasks.

W2. The paper lacks novelty, as applying the CLIP architecture in the e-commerce domain has been previously explored [a-c]. The authors need to further explain the innovation of their proposed method compared to existing methods.

We only use CLIP as a smart initialisation. Our key difference is that we perform early multimodal fusion of image/text modalities that allows us to obtain multimodal representations (text+image or category+image) without leveraging complex post-processing, whereas CLIP can only compute single modality representations that are comparable across modalities.

Regarding previous use of CLIP in e-commerce domain :

• FashionSAP is a fashion retrieval model, but it does not address CIS.
• FAME-ViL is a multi-task fashion model that includes CIR, but not RVS. We have added it to our baselines.
• e-CLIP is an adaptation of CLIP for event-based recognition, and we find it difficult to draw a direct connection with RVS, CIS, or fashion retrieval in general.

W3. The experiments require further refinement. Given that the authors utilized methods from composed image retrieval for performance comparison, it is essential to validate the proposed method on the FashionIQ and Shoes datasets within this context to effectively demonstrate its efficacy.

As mentioned earlier, we propose a benchmark for RVS that differs from CIR. We adapt some methods from the CIR literature to our setup, fully aware that they were not specifically tailored for our task. As expected, these methods do not yield strong results on our benchmark. Reciprocally, we do not expect our method, which is specifically designed for RVS, to perform well on CIR. In such tasks, specialization is key to attain good performances.

W4. The proposed method incorporates the Grounding DINO and SAM models; thus, a discussion on the fairness of performance comparisons with other methods is warranted. Additionally, the time and space complexity of the proposed method should be compared against other representative methods to provide a comprehensive evaluation.

Our method does not incorporate DINO nor SAM. They are only used to create a strong baseline (Table 1), which we are able to beat with our proposed architecture (Table 2). We agree that DINO and SAM are computationally heavy, but they are under-performing compared to our more frugal approach anyway.

References:

[1] Vaze, S., Carion, N., & Misra, I. (2023). Genecis: A benchmark for general conditional image similarity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

We thank the reviewer for their feedback and recognition of our efforts and address their concerns below.

W1. I am not convinced on the motivation of the proposed task. I cannot fully imagine a real application scenario for the proposed Referred Visual Search. According to my experience and understanding on fashion/ads, the query image (ie, dressed human) and the target image (ie, garment, product and so on) shown in Figure 1 are normally from the same product set which provided by the advertiser (at least they share some common information, eg, brand name). In that case, we don't need to do large-scale image-base retrieval, which is implementation-wise expensive in the industry. The only case I understand is letting the user themselves upload their individual query images and do a cross-domain image retrieval. However, the target and query image used in this paper are from the same domain and thus cannot model the real world problem, although I understand collecting cross-domain benchmark is even harder.

The intended use case for Referred Visual Search aligns with the second scenario: allowing users to upload a query image to retrieve an item from a large gallery.

However, collecting and utilizing natural user queries introduces significant ethical and privacy challenges. Datasets like LAION, for instance, have been taken down in the past for that reason. By focusing on professional images, we propose a safer alternative. While this approach does not perfectly replicate real-life scenarios, it serves as the closest and most durable proxy for establishing a reliable benchmark.

W2. I believe the proposed RVS is just a sub-task of Composed Image Retrieval (CIR), while their relationship is not clearly discussed (between L147 and L153) and the latest CIR related works were not cited. Regarding "rather than modifying the image" in L151, I think getting the feature of the ROI of the query image (Figure 4) is also kind of modifying the query image feature.

Our task belongs to the broader family of Conditional Image Similarity (CIS) tasks. While RVS is adjacent to CIR — as both fall under the CIS umbrella — it is not a special case of CIR.

As stated in GeneCIS [1]: “CIR [...] tackles the problem of composing an image and text prompt to retrieve relevant images from a gallery [...] with the text prompt acting as an image edit instruction (e.g. ‘the same dress but in white’). As such, CIR tackles a subset of the conditional similarity problem, by presenting models with a ‘negative’ similarity condition.”

Our RVS task is more closely aligned with the unnamed “Focus Object” task described in the GeneCIS benchmark, where retrieval is evaluated with a gallery size of 15 images. In contrast, our benchmark scales to a much larger gallery of 2M images, providing a much more realistic setup.

We have clarified the relationship between RVS and CIR in section 2 of the revised version of the paper.

W3. The proposed model arch is okayish but not that innovative (L081) to me. I agree this can be regarded as a strong baseline but don't think this widely used arch can be claimed as innovative.

Indeed, one key outcome of our paper is that our straightforward proposed method outperforms more complex approaches that rely on multiple dedicated detection modules. This result is both surprising and exciting as it shows that performance gains can be made by simplifying recommendation pipelines. Notably, our proposition — a soft conditioning of a vision transformer on a textual embedding — has not been explored in this field before.

We have clarified this claim in our revised version.

W4. I think some zero-shot methods (no training needed) should be considered as baselines too, eg, multimodal LLM that can fuse the query image feature and the instructions.

We added zero-shots baselines, including miniCPM-V 2.6 [2], a 8B SOTA VLM model. Since the VLM architecture is not inherently designed for image embedding, we experimented with different approaches, including global average pooling of the tokens or using only the last token from various layers.

We would like to emphasize that this baseline introduces a very heavy pipeline, with a significantly high computational cost for embedding the entire gallery (2M samples). Additionally, the higher embedding dimension increases both index size and retrieval latency.

The resulting method achieves 9.2% R@1 with 2M distractors, underscoring the necessity of developing more efficient, general-purpose conditional image retrieval models. These results are added in Table 2 of the revised version.

References:

[1] Vaze, S., Carion, N., & Misra, I. (2023). Genecis: A benchmark for general conditional image similarity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

[2] Yao, Y., Yu, T., Zhang, Ao., et al. (2024) MiniCPM-V: A GPT-4V Level MLLM on Your Phone. ArXiV

Dear reviewer,

Today is the last day for reviewers to ask questions to authors. Did the authors' rebuttal address your concern? Do you have any additional questions?

We thank the reviewer for their encouraging feedback and address their concerns below.

W1. While the experimental results are solid, it could be more comprehensive if the analysis can go deeper into the impact of objects' visibility in query images considering object sizes, occlusions or viewpoints. Or, a simpler referring aspect could be segmentation mask area or its ratio to a detected object bounding boxes? One example is how the retrieval accuracy changes among shirts because they are often partially visible due to overlaid with outwears.

We can indeed provide a more detailed analysis of our results by examining two additional metrics:

• R@1 aggregated by category. This uses the annotated categories from the test set.
• R@1 aggregated by mask area. For each test query, we compute a mask for the target object using Grounded SAM and manually simplify complex captions to ensure more accurate masks.

These metrics reveal that objects with low visibility — due to size, occlusion, or viewpoint — are more challenging to retrieve. We also observe a correlation with certain categories that predominantly contain small objects, such as gloves, shoes, and jewelry.

We have included this analysis in the revised version (see appendix B.4), as it provides valuable insights into the weaknesses of models with respect to specific aspects of the benchmark.

Q1. It would be interesting to discuss how to construct larger datasets in real-world scenarios and the challenges associated with generalizability to close domains such as street outfit photos that have noisy backgrounds, or even beyond fashion such as furniture, landmarks, etc.

We would like to emphasize potential ethical and privacy concerns when building large-scale datasets for real-world applications. In our study, we chose to focus on professional fashion images to avoid using personal data. While this approach may be slightly misaligned with certain real-world use cases, it represents the most durable and ethically sound method for creating a benchmark. This strategy could also be extended to other commerce categories, such as furniture.

With these considerations addressed, we identify two approaches for obtaining more realistic data for real-world applications:

1. Simulation: Leveraging generative models to create realistic contexts. This method avoids reliance on personal data but may have limitations in the realism of its outputs.
2. User-Generated Reviews: Utilizing user-taken photos associated with product reviews as noisy queries. This approach is feasible for large fashion retailers, provided it aligns with the Terms and Conditions of their services and regional legal settings (e.g., GDPR in the EU).

Dear reviewer,

Today is the last day for reviewers to ask questions to authors. Did the authors' rebuttal address your concern? Do you have any additional questions?

We thank the reviewer for their feedback and acknowledging the quality of our dataset and address their concerns below.

W1.1. The proposed task is like a special case for the task of composed image retrieval (CIR).

Our task belongs to the broader family of Conditional Image Similarity (CIS) tasks. While RVS is adjacent to CIR — as both fall under the CIS umbrella — it is not a special case of CIR. For further details on CIS tasks, we refer the reviewer to Vaze et al. [1]. We have clarified the relationship between our RVS task, CIS, and CIR in the revised version in section 2.

W1.2. However, the experiment is not compared with recent CIR methods, which makes the result not convincing.

We update the paper to include results from other pre-trained CIR and CIS models.

• MagicLens [2]: A SOTA generalist CIS model trained on 36.7M instructions triplets.
• FAME-ViL [3]: A recent fashion-specific CIR model
• MiniCPM-V 2.6 [4]: A SOTA Vision-Language model of 8B parameters.

Both MagicLens and FAME-ViL achieve close to 50% R@1 with 10K distractors, showing some degree of task transferability. However, their performance drops sharply with 2M distractors, reaching only 7.5% and 9.3% R@1, respectively. This emphasizes the specificity and difficulty of our fine-grained retrieval benchmark. These results are added in Table 2 of the revised version.

W2. The proposed method shows high performance in LRVS-F, as shown in Table 2, nearly 99% in Cat@1. These superior performances raise the question of the proposed dataset's motivation. Is there any specific problem the dataset wants to diagnose?

We report the Cat@1 as a sanity check. This metric does not adequately reflect a model's ability to retrieve the correct image, as an item could belong to the correct category without matching the object of interest in attributes such as color, pattern, or shape.

The most informative metric is R@1 with millions of distractors, as it measures the model's ability to retrieve an exact item from a large gallery. This metric highlights opportunities for improvement in future work. We have ensured these points are clearly explained in the revised version.

References:

[1] Vaze, S., Carion, N., & Misra, I. (2023). Genecis: A benchmark for general conditional image similarity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

[2] Zhang, K., Luan, Y., Hu, H., et al. (2024). MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions. Proceedings of the 41st International Conference on Machine Learning

[3] Han, X., Zhu, X., Zhang, L., et al. (2023) Fame-vil: Multi-tasking vision-language model for heterogeneous fashion tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

[4] Yao, Y., Yu, T., Zhang, Ao., et al. (2024) MiniCPM-V: A GPT-4V Level MLLM on Your Phone. ArXiV

Dear reviewer,

Today is the last day for reviewers to ask questions to authors. Did the authors' rebuttal address your concern? Do you have any additional questions?