Image2Sentence based Asymmetrical Zero-shot Composed Image Retrieval

We appreciate the detailed comments and acknowledgment of our contributions. We provide the responses as follows.

Q1: Since this work is a retrieval task, it is important to report how the retrieval performance varies as the gallery size scales up. I understand that the three evalutation datasets are standard benchmarks. However, it would make the contribution more solid if millions distractors could be involved in the gallery, although most existing SOTAs didn't report such results.

R1: We adopt the distractors from R1M in [4] to scale up the gallery set during evaluation. Note that for CIRR and CIRCO, the evaluation on test set could only be conducted on the official cloud server, thus we add distractors to gallery for the validation set. The results on three benchmarks are shown as below, which reveal that the distractors in the gallery would dampen the retrieval accuracy, however our method is still superior on three benchmarks.

Scaling up the gallery of FashionIQ validation set

Scaling up the gallery of CIRR validation set

Scaling up the gallery of CIRCO validation set

[4] Radenović, F., Iscen, A., Tolias, G., Avrithis, Y., & Chum, O. (2018). Revisiting oxford and paris: Large-scale image retrieval benchmarking. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5706-5715).

Q2: The inference time should be reported including in the query side and in the cloud side.

R2: We report the latency on both query side and cloud side on CIRCO vailidation set. For query side, we report the inference latency for different query models. We find that lightweight encoders are generally twice faster than large visual encoder (BLIP VE). For cloud side, since the gallery models are the same (BLIP TE & VE), the retrieval latency is the same.

Q3: Figure 1 could be further improved. Specifically, the pink rectangle and trapezoid could be shrunk and the blue trapezoids could be enlarged. As a result, it is much easier to quickly grasp the idea at first glance for readers.

R3: Thanks for the suggestions of Figure 1. We have revised Figure 1 for better clarity.

We appreciate the detailed comments and acknowledgment of our contributions. We provide the responses as follows.

Q1: The clarity of the writing, particularly in the methods section, requires improvement.

R1: Thanks for the suggestion, we have carefully revised our manuscript, with changes marked in blue for clarity. The major modifications include:

• the description of the framework in section 3
• the merits of adaptive token learner in section 3.1
• the clarification of inference phase in section 3.3

Please feel free to share if you have further concerns, we would be keen to address them as well.

Q2: For image-only retrieval, could you provide results using the DINO-V2 and MoCo-V3 pretrained models? The CLIP model is typically used for content matching between image and text features.

R2: We use the pretrained MoCo-V3 models (resnet50, ViT-base) and DINO-V2 models (ViT-base, ViT-large) to test the retrieval performance on three benchmarks. Note that self-supervised pretrained models are designed for visual modality and do not include the text encoder, thus we merely report the image-only retrieval results for them. For comparison, we also report the retrieval performance of our method that integrates both visual and textual information for retrieval.

The image-only results on FashionIQ validation set

The image-only results on CIRR test set

The image-only results on CIRCO test set

Generally, self-supervised models could improve the performance of image-only retrieval performance. However, we could see that they fail to make a difference for composed image retrieval compared with our method, since the CIR task requires comprehension of multi-modalities and the interaction between visual and textual information.

Q3: The transformation of the retrieval problem from image-to-image to text-to-image inherently assumes that the descriptive texts are of high quality and detailed. Any inaccuracies or vagueness in the text could lead to inefficient or incorrect image retrievals.

R3: As we mentioned earlier, different from "image-to-image" uni-modal retrieval that merely takes image as query, the task of composed image retrieval allows multi-modal queries to retrieve the images in the database, which is a task of "image + text → image". In practice, the text is given by the user to represent the accurate retrieval intent. Therefore, users could adapt the text modifier to be more detailed and precise to obtain more accurate search results.

In our framework, we convert the "image + text → image" problem to the "text-to-image" problem by mapping the visual information to the word embedding space of the text encoder in large foundation model as pseudo sentence. Therefore, the visual information could interact with the text modifier more efficiently in the text domain to obtain the multimodal composed features. We further utilize the text-image alignment capability of large foundation models to do accurate image retrieval, and achieve the best performance on three benchmarks.

Q4: The paper presentation is not very attractive. It is difficult to understand the novelties / contributions after reading the introduction of the paper.

R4: We are sorry for that we did not present clear contributions in the manuscript. Here we summarize our contributions as follows:

1. We propose a new framework for zero-shot composed image retrieval, which solves composition learning without the expensive labeled-triplets for supervised training.

2. We propose an asymmetrical structure for composed image retrieval to enable flexible deployment on resource-constrained platforms and improve the retrieval efficiency.

3. We propose an adaptive token learner to map image as a sentence in the word embedding space to interact with text information, and automatically filter out noisy visual information to preserve discriminative semantics.

4. Global constrastive distillation and local alignment regularization are proposed to guide the lightweight encoder learning with the knowledge of large foundation model, and our method achieves significant retrieval performance on CIRR, CIRCO and FashionIQ datasets.

Q5: How does the ZSCIR approach compare in performance and efficiency with state-of-the-art image retrieval methods that don't employ a text-to-image asymmetry? Are there scenarios where a traditional symmetric approach might outperform ZSCIR?

R5: As report in Table 2,3,4 in the manuscript, our asymmetrical method outperforms the state-of-the-art methods Pic2word and SEARLE in terms of retrieval performance, which adopt symmetrical structure. In Table 1 in the manuscript, we report the model parameter size and computational consumption for different query models. Here we consider Pic2word and SEARLE adopting BLIP VE as both query and gallery model. Compared with the symmetrical structure that adopts large visual encoder (BLIP VE), lightweight models require much fewer computational resources. Furthermore, we report the latency on both query side and cloud side on CIRCO validation set. For query side, we report the inference latency for different query models in the following table. We find that lightweight encoders are generally twice faster than the large visual encoder (BLIP VE). For cloud side, since the gallery models are the same (BLIP TE & VE), the retrieval latency is the same. Generally, the lightweight encoders are much more efficient than the heavy foundation model in terms of resource consumption and inference latency.

To train the symmetrical framework, the large visual encoder needs to be updated during training. Since the large visual encoder is much heavier than our lightweight encoder, it is more likely to overfit and requires larger GPU memory to reach the same batch size for good performance during training. With more training data and more GPU resources, we believe that the symmetrical setting would outperform the asymmetrical structure in terms of retrieval accuracy.

We appreciate the detailed comments and acknowledgment of our contributions. We provide the responses as follows.

We first make a clarification of composed image retrieval task and our method. As mentioned in Introduction (section 1) in the manuscript, the task of composed image retrieval allows multi-modal queries to retrieve the images in the database, i.e., the query includes an image, and a text that describes specified modification to the image. In other words, composed image retrieval is a task of "image + text → image". In practice, the text is given by the user to represent the accurate retrieval intent. Therefore, users could adapt the text modifier to be more detailed and precise to obtain more accurate search results.

For our method, we convert the "image + text → image" retrieval to the "text-to-image" retrieval by mapping the visual information to the word embedding space of the text encoder in large foundation model. During inference, the query image is converted as a pseudo sentence to concatenate with the text modifier, and they are together fed into the text encoder to extract the multi-modal query feature. The database image features are extracted by the visual encoder in large foundation model. Then by utilizing the text-image alignment capability of large foundation model, we are able to do accurate image retrieval with multi-modal query. Note that the asymmetry of our framework does not lie in the difference of modalities for the query and database side (text-to-image), but in adopting a lightweight model for the query side while adopting a large model for the database side to improve deployment flexibility and retrieval efficiency.

Q1: While the adaptive token learner is a strength in terms of converting visual features to textual tokens, there's a risk that the system could become overly reliant on this component. If the learner fails or encounters unanticipated scenarios, it might compromise the effectiveness of the entire retrieval process.

R1: In the context of zero-shot composed image retrieval, we do not have access to the labeled triplets (reference image, text modifier, target image) as direct supervision, which is necessary to train the multimodal composition capabilities of the model. Consequently, we convert visual information in the word embedding space as a descriptive pseudo sentence. This approach enables us to utilize the intrinsic composition ability of the text encoder, facilitating efficient interactions between visual information and the text modifier. This process is essential for extracting the multimodal query feature for retrieval. Therefore, it is critical to learn an effective adaptive token learner in our framework for the conversion, which is our main contribution. Our adaptive token learner is trained through global contrastive distillation loss and local alignment constraint with sufficient training data (3M), to align with the image feature space of BLIP model. Since BLIP model is pretrained on large-scale image-text dataset (129M), it possesses rich knowledge to guide the token learner to focus on prominent visual information and interact more effectively with textual information.

We admit that sometimes there will be difficulty for the adaptive token learner to catch the intended target in some complicated scenarios, as shown in the failure cases of Figure 6 in the appendix. Nevertheless, in most cases, our adaptive token learner could focus on the discriminative patterns and interact well with the text modifier as shown in the good example of Figure 5 in the appendix, which plays an important role in our framework to achieve outstanding retrieval performance, as shown in Table 2,3,4 in the manuscript.

In practical application scenarios, we can obtain a better token learner by enlarging the amount of training data and incorporating better multimodal foundation models to guide the learning process.

Q2: Introducing an asymmetric text-to-image retrieval approach, while innovative, adds an extra layer of complexity to the system. This might present challenges in terms of maintainability, debugging, and further development of the system.

R2: The lightweight visual encoders in our framework are much smaller and more efficient compared with the large foundation model, as they have fewer parameters and computations shown in Table 1. Therefore, it is much easier for maintainability, debugging and development compared with large foundation model. Although the asymmetrical structure brings about more complexity to the system, it enables faster inference for retrieval and deployment on resource-constrained platforms such as mobile phones or edge devices. Therefore, the asymmetrical structure helps to improve the retrieval efficiency. Moreover, the asymmetrical structure could also protect user privacy, since it only uploads the extracted tokens instead of real images on the Internet.

I thank the authors for writing very detailed rebuttal and reply all the weaknesses and questions that I mentioned. After looking into the rebuttal and also going through other reviews and replies, I think the paper is making a good contributions to the community. So I have decided to increase my rating. I hope the authors would address the comments in the final version of the paper.

We appreciate the detailed comments and acknowledgment of our contributions. We provide the responses as follows.

Q1: It could be better to discuss more about the number of token selections in detail. According to Figure 3, it seems the performance is a bit sensitive to the number of tokens used in the proposed method. Then, a more detailed discussion of this observation with a visual example of the same query but a different number of tokens could help to better demonstrate the impact brought by the tokens.

R1: Generally, to achieve accurate retrieval, the pseudo sentence tokens need to accomplish two aspects: extracting effective visual information and interacting with textual information. When the token length is very small, there would be drop in performance due to the lack of capacity to extract sufficient visual information. As shown in Figure 8 in the appendix of manuscript, query model with very small token length performs worse even if the attention maps are reasonable.

However, when the token length increases for too long, on one hand, some tokens may focus on trivial and noisy visual patterns; on the other hand, these tokens may interact incorrectly with the text modifier, and may impact the correct token-text interaction. In Figure 4 in the manuscript, we have demonstrated the comparison of different token lengths with the same query. With the visualization of the attention maps of adaptive token learner and the retrieval results, it could be seen that some sentence tokens may tend to focus on the background or the trivial patterns with excessively large token lengths, which would likely introduce noise to degrade the retrieval performance. We further show more comparisons of different token lengths, including the retrieval results, attention maps of sentence tokens with text modifier and the visual feature map in Figure 9 in the manuscript appendix. As shown in Figure 9, the excessive tokens associate the visual patterns with incorrect text information, potentially interfering with the interaction of multi-modal information. Even if there are accurate associations between other sentence tokens and text modifier, this interference would still degrade the retrieval accuracy.

Q2: It could be good to add a discussion of the relationship between the proposed adaptive token learner and the similar approach used in the following papers [1,2]. They have a similar structure, a discussion would help to better locate the position of the token learned in this work.

R2: In [1], the token learner is adopted to jointly learn local feature representation and aggregation in a unified framework. The output of token learner is the local features, which are concatenated as the aggregated global feature for image retrieval. In [2], the token learner maps a set of N input feature vectors to a set of K output vectors as slots, each of which may describe an object or entity in the input for the task of object discovery and set prediction.

As pointed out in [1] and [2], the token learner is utilized to focus on multiple discriminative visual patterns or entities and filter noise information such as background and indiscriminative image regions. Therefore, this structure could serve as an efficient visual extractor in various visual tasks.

Although our method also uses adaptive token learner, the semantics of token learner output are different from theirs. The output of our token learner resides in the word embedding space of the text encoder, which serves as a pseudo sentence, with each token acting like a word to interact with information of text modifier for further multimodal composition in CIR task. While in [1] and [2], the outputs are used directly for uni-modal vision tasks like similarity retrieval and object recognition. Therefore, the utilization of token learner in [1] and [2] is not suitable for our composed image retrieval task, while our token learner is devised for this task and achieves promising retrieval results in experiments.

[1] Wu H, Wang M, Zhou W, et al. Learning token-based representation for image retrieval. Proceedings of the AAAI Conference on Artificial Intelligence. 2022, 36(3): 2703-2711.

[2] Locatello, Francesco, et al. "Object-centric learning with slot attention." Advances in Neural Information Processing Systems 33 (2020): 11525-11538.

Finally, thank you again for your recognition and positive review to our work. We will incorporate your suggestions into our next revision.

Note: since OpenReview does not allow image uploading in the comment box, we place the additional figures in the appendix of revised manuscript.