CompoDiff: Versatile Composed Image Retrieval With Latent Diffusion

We thank the reviewer for their positive feedback and constructive comments. We will address all the raised concerns by the reviewer, and will revise our paper as soon as possible (Hopefully before Tuesday).

[W1] Meaning of diffusion for feature extraction

We would like to emphasize that the diffusion process itself is invariant to the input domain, whatever the input domain is, a diffusion model. Diffusion model just learns an underlying distribution of the given data into a Gaussian distribution. From this point of view, CompoDiff learns an underlying distribution of image latent embeddings with proper guidance. Note that our work is not the only method to learn a diffusion model on a latent space. For example, LatentDiffusion (or StableDiffusion) learns a diffusion model on 64 x 64 dimensional latent space of the VQ-GAN encoder. Dalle-2 prior model learns a diffusion model on CLIP latent feature space, as ours. If the reviewer wonders the difference between CompoDiff and these methods, please check our response for Reviewer AJF4, “CompoDiff vs. StableDiffusion”.

[W2] [W4] Clarity and typos in section 3.1 and Eq (4)

Thanks for your comment! We will revise our paper as soon as possible.

[Q1] Reliability of keyword-based diverse caption generation

Thanks for the question. Our process can handle the cases raised by the reviewer based on CLIP-based filtering. In page 14, we describe the details of our filtering process:

We apply a filtering process following Brooks et al. (2022) to remove the low-quality ⟨xiR , xc, xi⟩. We filter the generated images for an image-image CLIP threshold of 0.70 to ensure that the images are not too different, an image-caption CLIP threshold of 0.2 to ensure that the images correspond to their captions, and a directional CLIP similarity of 0.2 to ensure that the change in before/after captions correspond with the change in before/after images. Additionally, for keyword-based data generation, we filter out for a keyword-image CLIP threshold of 0.20 to ensure that images contain the context of the keyword, and for instruction-based data generation, we filter out for an instruction modified image CLIP threshold of 0.20 to ensure consistency with the given instructions.

Now, we explain how the examples by the reviewer can be handled by our filtering process. First, if the caption itself is really ridiculous, then StableDiffusion and Prompt2Prompt will not be able to generate proper images. For example, it could make a noisy pixels, just white images, or “broken” images for a weird caption. These images will be filtered out by CLIP similarity because we measure the similarity between the generated images is sufficiently high, and the broken images will have low similarities with clean images.

Second, we define keywords as “noun”; hence, different POS rather than noun will not be extracted. Similar to the first case, if the given caption is really weird and ridiculous to make an image, then the generated model will make a broken image. It will be filtered out.

[W3] Resource consumption

Thanks for your constructive feedback. We agree that this paper would be better to have a more careful discussion of resource consumption. We will include each item in the revised paper.

[W3-1] Complex training stage

We would like to emphasize that the two-staged training procedure is not too much complex strategy. For example, SEARLE employs two-staged training, where the first stage is for learning pseudo token embeddings, and the second stage is for learning the projection module by distillation using the learned pseudo token embeddings. Combiner also employs two-staged training, where the first stage is a pre-training stage for the text encoder and the second stage is for training the combiner module using the pre-trained model. Our two-staged training strategy is not a mandatory procedure, but we employ two-staged training for better performance. Conceptually, stage 1 is for a pre-training by training text-to-image generation diffusion model with pairwise relationships. The stage 2 is for a fine-tuning process using triplet relationships. Lastly, the alternative optimization strategy for stage 2 is for helping optimization, not for resolving the instability. As shown in Table 3, CompoDiff can be trained only with Eq (3), but adding more objective functions makes the final performances stronger. In terms of diffusion model fine-tuning, we argue that our method is not specifically complex compared to other fine-tuning methods, such as ControlNet.

We partially agree with the reviewer, that both stages are trained with massive data points. However, as shown in Table 4, the massive data points are not the necessary condition for training CompoDiff. In fact, Table 4 shows that CompoDiff follows scale law; namely, CompoDiff performances are consistently improved by scaling up the data points. As far as we know, this is the first study that shows the impact of the dataset scale on the zero-shot CIR performances.

[W3-2] Inference stage

We would like to emphasize that CompoDiff retrieval does not take a very long inference time, as the reviewer's concern. Below, we add the comparison table for the comparisons of the inference speed of comparison methods. The table will be added to the revised paper soon.

In the table, we can confirm that CompoDiff is practically useful with high throughput (about 230ms 120ms). Note that these numbers highly depend on the hardware. For example, in Table 5 in our main paper, we report the average inference time as 120ms with a better machine. In the above table, we measured the numbers with a bit worse machine than the machine used for Table 5. The previous numbers are based on batch size 32, while the other numbers are based on batch size 1. We fixed the table to avoid confusion

One of the advantages of our method is that we can control the trade-off between retrieval performance and inference time. Note that it is impossible for the other methods. If we need a faster inference time, even with a worse retrieval performance, we can reduce the number of diffusion steps. More detailed experiments for the trade-off are shown in Table 5.

Also, we would like to emphasize that one of our main contributions is a novel and efficient conditioning for the diffusion model. Instead of using a concatenated vector of all conditions and inputs as the input of the diffusion model, we use the cross-attention mechanism for conditions and leave the input size as the same as the original size. As shown in Table C.5., our design choice is three times faster than the naive implementation.

[W3-3] OPT-6.7B model fine-tuning

We fine-tune the OPT model using LoRA fine-tuning (Low-Rank Adaptation of Large Language Models) with 8-bit quantization. We would like to emphasize that the quantized LoRA fine-tuning is very lightweight, only a single GPU can handle the model and the fine-tuning usually takes less than one day. Note that InstructPix2Pix uses the GPT-3.5 turbo API, which needs additional API cost, but our LoRA fine-tuning is easier, cheaper, and faster.

Dear Reviewer e76K,

Thanks for your constructive and valuable comments on our paper. We would like to notify the reviewer that the revised paper has been uploaded. The revised contents are highlighted in magenta. We have revised our paper to address the reviewer's concerns.

• [W1] Meaning of diffusion for feature extraction: We add the background section of diffusion models in Section 2.1. Due to the page limitation, the detailed discussion is in Appendix A.2, including our response to the reviewer's question.
• [W2] [W4] Clarity and typos in section 3.1 and Eq (4): We fixed the typo. We also omit the time embedding in Figure 3, where the time embedding is too fine detail compared to other concepts. Instead, we specify that the time embedding is a common choice for the existing diffusion models in Appedix A.2
• [Q1] Reliability of keyword-based diverse caption generation: We clarify that there exists an additional filtering process in Section 4, and split previous B.3 ("Triplet generation from caption triplets") to B.3 ("Triplet generation from caption triplets") and B.4 ("CLIP-based filtering") for emphasizing the filtering process. We also added our response to the reviewer's concern in Section B.4.
• [W3-1] Complex training stage: We clarify the training complexity of our method is not specifically complex compared the others in Section C.1. Also, to avoid confusion, we revise the sentence "Due to the training stability, CompoDiff uses a two-stage training strategy" to "CompoDiff uses a two-stage training strategy" and clarify the meaning of each stage instead. Note that it is not for hiding our weakness, but it is to avoid confusion such as, "our method shows unstable training (hence, not converged) without two-stage training" which is not true.
• [W3-2] Inference stage: We add Appendix D.1 Inference time comparison. Note that our first response has an error: CompoDiff takes 0.12s for forwarding a single image, while 0.23s is for batch size 32.
• [W3-3] OPT-6.7B model fine-tuning: We clarify that the LoRA fine-tuning is lightweight in Section 4. We also would like to emphasize that OPT is only for generating the synthetic training dataset, not for the retrieval model.

Please feel free to ask anything to us if the reviewers think the revised paper is insufficient. We are open to discussion and will address all the concerns of the reviewers.

We thank the reviewer for their positive feedback and constructive comments. We will address all the raised concerns by the reviewer, and will revise our paper as soon as possible (Hopefully before Tuesday).

[W1] More analytical reasoning of the CIR benchmark performance gaps is encouraged

Thanks for the comment! First, we would like to emphasize that our experimental results are “zero-shot”, which means the models are not trained on the target dataset. It can cause a significant domain gap between the target dataset domain and the training dataset. CIR datasets, such as FashionIQ and CIRR datasets, have specific domain characteristics that cannot be solved without training on the datasets. For example, the real-world queries that we examine are mainly focused on small editing, addition, deletion, or replacement. However, because the datasets are constructed by letting human annotators write a modification caption for a given two images, the text conditions are somewhat different from the real-world CIR queries. For example, the CIRR dev set has some text conditions like: “show three bottles of soft drink” (different from the common real-world CIR text conditions), “same environment different species” (ambiguous condition), “Change the type of dog and have it walking to the left in dirt with a leash.” (multiple conditions at the same time). These types of text conditions are extremely difficult to be solved in a zero-shot manner, but we need access to the CIRR training dataset.

When we perform a qualitative study on LAION-2B image index, we observe that the retrieval quality becomes better by increasing the dataset scale from 1M to 18.8M. We could not build a quantitative evaluation benchmark on the LAION-2B index set, because as our previous comment, the existing datasets need human verification, and it is impossible to perform on billion-scale images. The example retrieval results from LAION-2B are shown in Fig 6 and C.1.

[Q1] Does this retrieval include images containing multiple target objects for retrieval as well?

CIRR partially contains such tasks, for example: “Change the type of dog and have it walking to the left in dirt with a leash.” (multiple conditions at the same time). However, basically, as CIRR benchmarks are built upon captions written by human annotators, there are no specific subsets or benchmarks to target multiple target objects. Conceptually, CompoDiff can handle multiple target objects if the target is given in a sentence (the same as the other methods). If target conditions are given in multiple sentences, we can iteratively edit the latent feature, i.e., orig_feature -> (CompoDiff) -> edited_feature_by_sent_1 -> (CompoDiff) -> edited_feature_by_sent_2. However, we do not have a proper quantitative benchmark to compare the methods in such scenario.

[W2] [W3] Enhance writing quality: backgrounds and training paradigm

Thanks for your suggestion. We agree that the current manuscript can be a bit difficult to understand without enough background. We will revise the paper in the rebuttal period. We will inform the reviewer when we upload the revised paper.

[W4] [Q2] Learnable text prompts

Thanks for the suggestion. We believe that the intuition behind this comment means that “Will the quality of the text embedding affect to the final retrieval performances?”, and the answer is “yes”. It can be proven by our experiments (Table 6) on changing the text encoder for the condition of CompoDiff diffusion model. In the table, we can observe that using a better text encoder boosts up the CIR performance with a large gap (e.g., 38.20 -> 44.11 for FashionIQ recall and 29.19 -> 39.25 for CIRR recall). It means that if we can use better text information, the overall performance will be improved.

However, unfortunately, it is not straightforward to apply learnable text prompts to our method. It is because our text encoder is not used for feature matching as other learnable text prompt literature, but the text embedding is used for the cross-attention of the diffusion model, and the supervision for the classifier-free guidance of our diffusion model. It is not trivial to apply learnable prompts to such scenario, especially for classifier-free guidance. We have searched related works, but we cannot find any sound method to be applied to CompoDiff. If the intuition behind this question is the relationship between the text information quality and the retrieval performance as our assumption, we hope our experimental result can answer the question.

We appreciate the reviewer for their positive comments and constructive feedbacks. We will address all the raised concerns by the reviewer, and will revise our paper as soon as possible (Hopefully before Tuesday).

[W1] The authors just proposed a larger dataset

Our main claim is the problem of requiring pre-collected and human-verified triplets, not triplets themselves. We will tone down our argument in the introduction section as: “obtaining human-verified high-quality triplets can be costly”. We will revise our paper as soon as possible. Meanwhile, we would like to emphasize that the dataset scale-up is one of our main contributions: it is challenging to make a synthetic dataset that shows the scaling law, i.e., more data points lead to better performances. Our dataset construction process achieves this property by employing the keyword-based caption generation that ensures the diversity of the captions. As shown in Table 4, the previous data generation process by IP2P shows inferior model performance than our generation process with the same scale (27.2 vs. 31.9 in the Fashion IQ average recall). We will discuss more details in the next bullet.

[W2] The value of the dataset is limited to a certain extent

We would like to emphasize that the scale shown in Table 4 is already very large. Note that 1M is the scale of ImageNet, and 10M is ten times larger than ImageNet. It is even larger than ImageNet 21K (11.8M images). Scaling up the dataset from 10M to 18.8M shows a bit saturated performance improvements in the benchmark, but we would like to argue that the benchmark number is somewhat noisy in CIR. When we perform a qualitative study on the LAION-2B image index, we observe that the retrieval quality becomes better by increasing the dataset scale from 10M to 18.8M. The reason why the given benchmark scores look saturated is that FashionIQ and CIRR datasets have specific domain characteristics that cannot be solved without training on the datasets. For example, the real-world queries that we examine are mainly focused on small editing, addition, deletion, or replacement. However, because the datasets are constructed by letting human annotators write a modification caption for a given two images, the text conditions are somewhat different from the real-world CIR queries. For example, the CIRR dev set has some text conditions like: “show three bottles of soft drink” (different from the common real-world CIR text conditions), “same environment different species” (ambiguous condition), “Change the type of dog and have it walking to the left in dirt with a leash.” (multiple conditions at the same time). These types of text conditions are extremely difficult to be solved by a zero-shot manner, but we need to access to the CIRR training dataset.

We are not certain about the “other datasets” from the comment “The experimental results show that the effect of the proposed data set and other data sets are similar at the same level”, but if other datasets denote the CIR datasets, such as CIRR or FashionIQ training set, we would like to emphasize that it is not a fair comparison with ours. We aim to solve “zero-shot” CIR, i.e., without accessing the expensive human-collected triplets. Directly comparing our zero-shot CIR results and the supervised CIR results on the target dataset is not fair. Note that CLIP ViT-B/16 ImageNet zero-shot classification performance is 80.9 (where the training dataset scale is 400M), worse than that of DeiT-B/16 (a supervised method on 1.2M ImageNet training set) 84.2, but we do not argue that the value of CLIP zero-shot performance is negated.

[W3] Inconsistent training dataset for comparison methods / [Q1] Additional comparison experiments for the comparison method are trained on SynthTriplets18M

We already fit the training datasets for other triplet-based fusion methods, such as ARTEMIS and Combiner. For a fair comparison in terms of backbone size, we have conducted additional experiments with CompoDiff with the same backbone:

(cont.)

Dear Reviewer FhVu,

Thanks for your constructive and valuable comments on our paper. We would like to notify the reviewer that the revised paper has been uploaded. The revised contents are highlighted in magenta. We have revised our paper to address the reviewer's concerns.

• [W1] The authors just proposed a larger dataset: We clarify that previous CIR methods are based on pre-collected human-verified triplet dataset in the Introduction. We also emphasize that human verification is not scalable, while our dataset is easily scalable automatically to an infeasible scale for manual collection (e.g., more than 1M). The related discussions are in Section 5.3, Appendix C.3.
• [W2] The value of the dataset is limited to a certain extent: We add a related discussion in Section 5.3, Appendix C.3. Particularly, we add a discussion of why the existing benchmarks are somewhat unreliable benchmarks for evaluating authentic zero-shot CIR performances in Appendix C.3. We also clarify that our main evaluation is zero-shot, while FashionIQ-trained counterpart is fully supervised in the Introduction.
• [W3] Inconsistent training dataset for comparison methods / [Q1] Additional comparison experiments for the comparison method are trained on SynthTriplets18M: We clarify that it is impossible to train Pic2Word and SEARLE in Section 5.1 and Appendix C.3. We also added CompoDiff RN50 results for a fair comparison with other RN50-based methods on SynthTriplets18M.
• [Q2] Impact of the dataset scale to the other methods: We add the experiments and discussions in Appendix D.4 (Table D.8)

Please feel free to ask anything to us if the reviewers think the revised paper is insufficient. We are open to discussion and will address all the concerns of the reviewers.

Thanks the authors to provide additional experimental results. From the added results, the performance of the models is improved clearly when the scale of the dataset is expanded from 1M to 10M. However, there is only a slight improvement in performance when the scale of the dataset is expanded from 10 M to 18 M, which implies that it is may be unnecessary to expand the scale of the dataset to 18M. In addition, the proposed method requires the mask of the object in the reference image as a condition for training, which results in the model hardly being trained on other datasets. Therefore, I would keep my rating on the post-rebuttal section.

We thank the reviewer for their positive feedback and constructive comments. We will address all the concerns raised by the reviewer, and will revise our paper as soon as possible (Hopefully before Tuesday).

[W1] Comparison with the same backbone

Thanks for the comment. We would like to emphasize that CompoDiff has the same backbone architecture (e.g., ViT-L) as Pic2Word and SEARLE, which are designed for zero-shot CIR. We have conducted CompoDiff with RN50 to compare with ARTEMIS and Combiner with the same backbone as the reviewer suggested. The table below shows that in most of the metrics, CompoDiff still outperforms the comparison methods with large gaps. For example, CompoDiff RN50 shows 18.02 CIRR R@1, while ARTEMIS and Combiner show 12.75 and 12.82, respectively. If we scale up the backbone to ViT-L and ViT-G, the scores become 19.37 and 26.71, respectively. It shows that the improvement by CompoDiff is not by a larger backbone but by our novel diffusion-based retrieval strategy. We will include the full results in the paper.

In addition, we would like to emphasize that our main contribution is two-fold: (1) the diffusion-based CIR model and (2) a new synthetic triplet dataset. We believe that a weakness in the first contribution does not harm the second one.

[W2] Efficiency comparison

Thanks for the suggestion. We agree with the comment. Below, we add the comparison table for the comparisons of the inference speed of comparison methods. The table will be added to the revised paper soon.

In the table, we can confirm that CompoDiff is practically useful with high throughput (about 230ms 120ms). Note that these numbers highly depend on the hardware. For example, in Table 5 in our main paper, we report the average inference time as 120ms with a better machine. In the above table, we measured the numbers with a bit worse machine than the machine used for Table 5. The previous numbers are based on batch size 32, while the other numbers are based on batch size 1. We fixed the table to avoid confusion

One of the advantages of our method is that we can control the trade-off between retrieval performance and inference time. Note that it is impossible for the other methods. If we need a faster inference time, even with a worse retrieval performance, we can reduce the number of diffusion steps. More detailed experiments for the trade-off are shown in Table 5.

Also, we would like to emphasize that one of our main contributions is a novel and efficient conditioning for the diffusion model. Instead of using a concatenated vector of all conditions and inputs as the input of the diffusion model, we use the cross-attention mechanism for conditions and leave the input size the same as the original size. As shown in Table C.5., our design choice is three times faster than the naive implementation.

Dear all the reviewers,

Thanks for your time in reviewing our paper. We have addressed all the raised concerns from the reviewers with the revision of the texts and additional experiments (CompoDiff RN50, ARTEMIS and Combiner experiments by varying the dataset scale, and the inference time comparison). It will be a pleasure if the reviewers can re-evaluate our paper based on the revised paper.

As the revision period is closing in 15 mins, we will not able to timely update our manuscript, but we will address all the concerns from the reviewers in the final revision.