Sentence-level Prompts Benefit Composed Image Retrieval

We thank the reviewer for his/her constructive comments and provide our point-wise replies as follows.

W1: How does sentence-level prompts work like in Fig.1(c).

A1: Yes, our sentence-level prompts are latent word embeddings. Nevertheless, the sentence-level prompts in our method are a sequence of word embeddings, although not in human-readable form, which is continuous and beneficial for learning. While the purpose of Fig. 1 (c) is to visualize our motivation.

To further prove whether our method can achieve the effect shown in Fig. 1 (c), we trained SPRC on CIRR and the zero-shot tests on the GeneCIS data demonstrate the efficacy of SPRC in capturing and modifying objects and attributes, i.e., the metrics of Change Attribute, focus attribute, and Change Object in the table below, etc.

W2: How the prompt alignment loss works.

A2: Sorry for the confusion. We use the same auxiliary text prompt for all reference images, therefore, '$p_i^\prime$' should be '$p^\prime$'. We have revised it in our revised manuscript.

In our method, two types of loss functions, i.e., ITC loss and alignment loss, are utilized to jointly optimize the sentence-level prompt $p_i$.

Specifically, the alignment loss indicates calculating the L2 distance between $p_i$ and $p'$.

We note that during the alignment of $p_i$ to $p'$, the gradient of $p'$ does not need to be computed.

During the optimization of $p'$, the text encoder related to $p'$ is not entirely frozen but is aligned through the text encoder trained by p using an EMA approach. At this stage, the image encoder remains frozen. Therefore, during the optimization of $p'$, we only need to learn $p'$, as the text encoder copies weights via EMA, and the image encoder is frozen. Hence, the optimization of $p'$ does not affect any parameters of the text encoder, image encoder.

In summary, the primary role of $p'$ is to act as a constraint for the sentence-level prompt $p_i$, ensuring that $p_i$ can align more closely with the word embedding feature space.

To verify the effect of the alignment loss, we compared the performance of the model on the CIRR and F-IQ validation sets in Tables 6 and 7 of the supplementary material. The results show that the smaller the weight of the alignment loss, the worse the performance of the model. This further verifies the effectiveness of the alignment loss, and we also provide the performance without the alignment loss as follows:

As the discussion period draws to a close soon, we extend our sincere gratitude to the reviewer for his/her valuable time and insightful feedback. We value the constructive feedback and hope that our responses have appropriately addressed all the concerns.

We really appreciate the valuable time to respond to our feedbacks based on the reviewer's comments. Further, we are happy to address any remaining concerns.

We thank the reviewer for his/her constructive comments and provide our point-wise replies as follows.

W1: Innovation moderate; open to persuasion.

A1: The key innovations of our research include:

(1) Our work enhances the expression capability and accuracy of CIR by learning sentence-level prompts from reference images and relative captions. This approach is contrasted with the typical late fusion strategies or pseudo-word token generation in existing models, providing an effective solution for complex scenarios such as object removal and attribute modification in CIR.

(2) The effectiveness of our method is not limited to a specific retrieval model; it exhibits consistent superior performance across different models such as CLIP, BLIP, and BLIP-2.

W2: Cross-domain adaptability of SPRC.

A2: To verify the domain generalization ability of SPRC, we conducted zero-shot tests on CIRCO, GeneCIS, ImageNet (consists of ImageNet [1] and ImageNet-R [2]). ImageNet consists of samples from several different domains (cartoons, origami, toys, and sculptures), refer to the table below for details.

The results in the tables below show that the zero-shot performance of our method (pre-trained on CIRR) on the CIRCO is significantly better than the previous zero-shot method SEARLE [3].

Our method also surpasses the previous supervised method Combiner (Baldrati et al., 2022a) on GeneCIS dataset. This clearly shows that our model can adapt to domain shifts.

Similarly, using CIRR training and conducting zero-shot tests on ImageNet also achieved noticeable improvements against Combiner.

Although there is a clear domain difference between Fashion-IQ and other data, our method still obtained considerable performance in zero-shot tests trained on the Fashion-IQ dataset, see Table 1, 3. This result further shows that our model has the ability to handle domain shifts.

[1] Deng, Jia, et al. "Imagenet: A large-scale hierarchical image database." 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009.

[2] Hendrycks, Dan, et al. "The many faces of robustness: A critical analysis of out-of-distribution generalization." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[3] Baldrati, Alberto, et al. "Zero-Shot Composed Image Retrieval with Textual Inversion." arXiv preprint arXiv:2303.15247 (2023).

As the discussion period draws to a close soon, we extend our sincere gratitude to the reviewer for his/her valuable time and insightful feedback. We value the constructive feedback and hope that our responses have appropriately addressed all the concerns.

We really appreciate the valuable time to respond to our feedbacks based on the reviewer's comments. Further, we are happy to address any remaining concerns.

We thank the reviewer for his/her constructive comments and provide our point-wise replies as follows.

W1: Difference with https://arxiv.org/pdf/2310.09291.pdf.

A1: Thank you for sharing the CIReVL paper (Karthik et al, (2023). However, we would like to note that CIReVL is a concurrent work with our SPRC, and the arXiv upload date of this paper, i.e., October 13th, is after the submission deadline for ICLR 2024. As suggested, we performed our SPRC on a series of zero-shot tests on the CIRCO test set for comparison with CIReVL (we also discussed the CIReVL in our revised manuscript). The results are as follows:

The results in Table 1 show that, when pre-trained on the CIRR dataset and tested for zero-shot performance on the CIRCO dataset, our SPRC significantly outperforms CIReVL across all metrics. The domain differences between the two datasets, CIRR and CIRCO, indicate the generalization ability of our proposed model.

Furthermore, from Table 2, zero-shot test results on the GeneCIS test data show that our SPRC model performs better compared to CLReVL.

Additionally, while CLReVL outperforms our SPRC model when tested using ImageNet data, i.e., Table 3, SPRC outperforms the baseline model Combiner (Baldrati et al., 2022a).

W2: Results of CIRR are val. or test set?

A2: Thanks for the comment. We would like to clarify that all our experiments follow the settings of previous CIR works. For example, the results recorded in Table 2 from our main manuscript are all from the test set of CIRR, while the validation set is only used for ablation studies. This is consistent with previous CIR works (Liu et al., 2023b, Liu et al., 2023a). We also emphasized this point in the revised manuscript.

Q1: Potential to enrich the caption.

A3: Actually, the sentence-level prompts in SPRC are implicit word embeddings rather than explicit words; hence, SPRC cannot explicitly generate real text descriptions to enrich the dataset. However, SPRC is orthogonal to CIReVL, which can enrich data. For example, the target captions generated from CIReVL may be utilized as supervision to train SPRC's prompts to obtain better generalization abilities.

Q2 Extensibility of SPRC to Other domains/modalities.

A4: As shown in Table 3, when we perform zero-shot inference on ImageNet data, our model outperforms the previous SOTA zero-shot methods, i.e., Pic2Word (Saito et al., 2023) on Cartoon and Origami domains and obtains comparable performance on other domains. This showcases our model's ability to address different domains. We also observed that CIReVL outperforms our model, indicating that the explicit text prompts generated by the CIReVL model are beneficial for processing data across different domains. This potentially complements the sentence-level prompts in our SPRC, which can enable SPRC to achieve better performance across various domains.

As the discussion period draws to a close soon, we extend our sincere gratitude to the reviewer for his/her valuable time and insightful feedback. We value the constructive feedback and hope that our responses have appropriately addressed all the concerns.

We really appreciate the valuable time to respond to our feedbacks based on the reviewer's comments. Further, we are happy to address any remaining concerns.