Vision-by-Language for Training-Free Compositional Image Retrieval

We thank the reviewer for their detailed comments, and the appreciation of our flexible and interpretable CIR system, thorough experimentation and clear visualization. We address their issues and feedback below.

1. Limited Contribution:
We agree with the reviewer that our approach is a combination of existing VLMs and LLMs. However, their particular recombination to transfer it to compositional image retrieval, in which previous works have always leveraged (large-scale) additional training data to make the system work, are to the best of our knowledge, novel insights. In addition to that, the recombination and particularly chosen modular structure of our proposal offers an approach to CIR which is i) inherently interpretable unlike existing methods (which also leverage existing pretrained VLMs & LLMs), ii) can be easily scaled up and iii) allows for human intervention into the retrieval process - all properties, which are not existent in previous works which require additional training data! As such, we do believe that our paper provides a novel, practical contribution on compositional image retrieval.

2. & 3. Typo in the result and errors in table formatting:
We thank the reviewer for pointing out these errors, and have rectified these in the updated draft.

4. On the differing impact of captioning and LLM modules:
The reviewer rightly points out that particular choices in the image captioning module have less impact on the overall compositional image retrieval process as opposed to changes in the reasoning module (the choice of LLM). We do believe this to be primarily due to two reasons. First, the performance gaps between the tested captioning models is much less evident compared to the gap between available foundational LLMs (Llama versus GPT-4 have vastly different reasoning capabilities). This transitions to the second aspect, in that - as correctly noted - for all benchmarks, only a subset of crucial image properties have to be captured. This can be sufficiently done by all captioning models. Indeed, when looking at failure cases such as in Fig. 3, we find that a lack of captioning quality to be one of the main error sources - something which a notably better captioning model could fix in the future.

Given the modularity of our setting, something that can be simple hot-swapped, as opposed to having to retrain a full pipeline from scratch! Finally, we believe that a more detailed investigation should involve the development of benchmarks which cannot be easily described through single, in parts simplistic captions. This is something we leave for future work to tackle.

5 & 6 & 7. Writing Suggestions, missing references in 3.2 and missing baseline references:
We thank the reviewer for their valuable feedback. We have updated our paper accordingly, by adding a more detailed breakdown of the compositional image retrieval task, as well as referencing related ZS-CIR methods in the beginning of Section 3.2 (particularly Pic2Word & SEARLE). In addition, we have included the “text-only”, “image-only” and “image+text” baseline across our experimental results. We do note that the interpretation of our results is not impacted by this additional inclusion, though agree that it belongs there for completeness. Finally, for our GeneCIS experiments, we have included information about our utilized backbone network, which follows the one utilized in the original GeneCIS paper.

8. Limitations: The reviewer correctly points out that while our approach offers several crucial benefits such as being high-performant, training-free, flexible and scalable, interpretable and intervenable, there is no free lunch, and a certain trade-off is to be expected. We will provide additional information for each limitation noted by the reviewer individually.

• On inference cost & efficiency compared to previous works: Indeed, while existing methods also utilize large VLMs, the use of distinct captioning and reasoning modules elicits additional runtime. On our hardware setup (V100 / A100 GPUs) when compared to trained methods, we find inference times to increase by ~1s. However, the modularity also allows our setup to directly benefit from efficiency changes within each module (as e.g. seen in the cost & inference time reductions for LLMs over the past year), allowing one system to remain operational for a long period of time. This stands in particular contrasts to other ZS-CIR methods that require costly training to incorporate any pipeline changes or shifts in expected datasets (e.g. Pic2Word trains the mapping network on 8 GPUs on the complete CC3M dataset).

With the discussion period coming to an end, we would like to thank the reviewer again for their helpful feedback, and hope that our replies, alongside strong additional experimental results and an updated draft have addressed all questions. Of course, we are happy to continue the discussion in case of any further concerns.

We appreciate the response by the reviewer, and thank them for appreciating our work.

We do have extensively discussed issues raised by the reviewer, and have uploaded an updated draft, which has polished the writing based on the reviewers suggestions, while also including comprehensive additional, very positive results on Fashion-IQ.

We also would like to note that a discussion on limitations is also available (see response), where we showcase possible failure cases (alongside the option to address them through interventions) or have experiments that showcase compatibility of different modules.

If there is any other discussion the reviewer would like to see included and they believe has not yet been addressed, we would gladly do so.

We thank the reviewer for their thoughtful comments. We appreciate that they find our results promising, and our method interpretable. Below, we address the concerns raised:

1. Novelty and results on CIRR:
Indeed, our method is an application of off-the-shelf VLMs and LLMs. However, their particular recombination and reuse to transfer it to the difficult task of compositional image retrieval is novel. This is particularly interesting, as unlike previous approaches that leverage off-the-shelf VLMs as well and have to train on large amounts of data, our approach requires no training at all, while the modular structure and operation in the language domain makes our approach interpretable and intervenable as well.

This strongly contrasts our approach from existing ones! In addition, as our scaling laws study shows, our setup can be easily scaled up to account for modular improvements over time, without having to retrain. As such, we strongly believe that our work is an important contribution to the field of compositional image retrieval. In addition, we outperform existing, trained methods on CIRCO and Fashion-IQ significantly, while on CIRR (with noisy supervision, lack of multiple positive annotation, etc. as shown in previous works and noted in the paper), we are able to match previously obtained results

2. Results on Fashion benchmarks:

We thank the reviewer for the suggestion. We have now updated our paper with the results on the Fashion-IQ benchmark. For the detailed results, we refer to our shared comment. In summary, we find that CIReVL significantly outperforms existing methods on all three subtasks, e.g. improving average Recall@10 by nearly 6pp (28.29% versus 22.89% for SEARLE) with a ViT B/32 backbone. In addition, the simple scalability allows us to hot-swap the retrieval model to achieve additional 4pp (increasing from 28.29% to 32.19%).

3. Inference time:
Indeed, while existing methods also utilize large VLMs, the use of distinct captioning and reasoning modules elicits additional runtime. On our hardware setup (V100 / A100 GPUs) when compared to trained methods, we find inference times to increase by ~1s. However, the modularity also allows our setup to directly benefit from efficiency changes within each module (as e.g. seen in the cost & inference time reductions for LLMs over the past year), allowing one system to remain operational for a long period of time. This stands in particular contrasts to other ZS-CIR methods that require costly training to incorporate any pipeline changes or shifts in expected datasets (e.g. Pic2Word trains the mapping network on 8 GPUs on the complete CC3M dataset).

We thank the reviewer for their response, and would like to address two mentioned points:

(...) the novelty of this paper is not significant enough"

It would be great if the reviewer could provide additional context for this statement, as in our reply, we do highlight that our method and framework is novel. Even if the single components themselves are not novel on their own, their particular recombination and reuse is. Other approaches also leverage large-scale, off-the-shelf pretrained models and existing network architectures. Unlike these approaches however, our setup does not require large amounts of additional training data on top.

In addition to that, our method is first and foremost motivated by important practical constraint listed above, which significantly differentiates it from existing approaches, namely it being

• interpretable,
• intervenable,
• easily extendable,
• and providing better performance.

In addition to that, it also provides an important sanity check for future method that do incorporate additional training data.

(...) the performance and inference time of the proposed method still limits its real application

Large-scale models find ubiquitous usage in real world applications, ranging from LLM services to recent large-scale multimodal models. In addition, our performance significantly outperforms existing methods on all benchmarks but one (and comparable scores in that case), while not requiring any training. Furthermore, the ability to freely interchange models is a property that especially allows for more realistic deployment based on cost-performance trade-offs.

We thank the reviewer for their thoughtful comments. We appreciate that they found our paper ingenious, while highlighting the particular benefits with respect to interpretability and the capacity to allow for interventions. We address the concerns below:

1. Is a caption sufficiently descriptive?
We agree that a simple textual caption is limited in its ability to fully reflect the content of an image. This is something we also specifically highlight throughout the paper, particularly when addressing failure cases (Fig. 3) and their addressal through interventions in Fig. 4.

Generally however, when compared to current methods for Zero-shot Compositional Image Retrieval (ZS-CIR) which focus on mapping an input image to a single token in the textual embedding space, CIReVL is far less restrictive - generating a 10-20 word caption which can leverage e.g. 50+ tokens to represent the information in the image.

In addition, operating in the language space allows our approach to be inherently interpretable, and actually intervenable. Given these benefits, our work particularly tries to emphasize how far one may push a language-centric compositional image retrieval approach, and successfully shows that compared to existing methods, current benchmarks can be effectively tackled by purely representing each image with a caption/description.

Future work should focus on developing benchmarks that have information that cannot be easily described with a single description. In the current benchmarks, these cases are not the majority, while we do illustrate a potential example in Fig. 3.

2. Differences to Visual Programming:
While related, there are several significant differences between our proposed approach and Visual Programming. Visual Programming aims to solve a variety of vision and language tasks at the same time using an LLM to control the APIs to various modules.

However, all the tasks tackled by Visual Programming typically relate to working with a single image at a time, and don’t account for large-scale applications in which e.g. retrieval over an expansive gallery set has to be performed. To possibly adapt the Visual Programming method to this task, one would need to precompute the image features for all the images in the gallery set, provide sufficient in-context examples and a task description to enable the LLM to reason over the description, and provide particular instructions on how to find the correct final image. All of these modifications involve significant specialization to the CIR task and move the end result far away from the Visual Programming generalist.

In general, VisProg is a general framework which leverages different specialist modules and figures out ways to recombine them. Consequently, CIReVL should be regarded more as an expert, interpretable compositional image retrieval module that can then be leveraged by generalist agents via e.g. VisProg.

With the discussion period coming to an end, we would like to thank the reviewer again for their helpful feedback, and hope that our replies, alongside strong additional experimental results and an updated draft have addressed all questions. Of course, we are happy to continue the discussion in case of any further concerns.

We thank the reviewer for their detailed reply and reference, and would like to note two things:

Textual inversion is an optimisation-based approach that (to some extent) guarantees the text token is "faithful" to the image with respect to the image retrieval task. The same cannot be said about the image captioning model that "looks" at an image and generates a 50-word caption.

If e.g. looking at SEARLE, the textual inversion network is trained to generate a token that simply mimics classnames/concepts - so information available through the generated token will still be very lackluster. In addition to that, the computed text token is then incorporated into a full input prompt which is then fed into a text embedding system - and there is only so much information that can be packed into a single text token to account for the entire CIR task.

Taken together, a lot of relevant context will still go missing - and without a clear mechanism for interpretability, it is unclear what exactly.

On the other hand, there is no limit to the captions we utilize, and as the reviewer rightly highlighted, the captioning system can be extended arbitrarily (if needed), without impacting the overall pipeline. The generated captions generally provide much more context (as shown also qualitatively in the paper) beyond a single concept, and the resulting high performance gains we achieve - even when using the same retrieval system - support this.

While the captioning module may be super accurate, it can miss some "less important" details that I want to change. Fig-4 seems like a patchwork to me -- if a human has to modify the generated image caption, he/she might as well write it in the first place.

This is of course true - the captioning system is not perfect, but given the modular nature, can simply we replaced in a plug-and-play fashion if needed. However, we show that even with very general open-source captioning models, that very strong CIR performance can be achieved.

Regarding the human just writing the caption themselves, this is exactly the point of the intervenability, and a byproduct of our inherent interpretability. If the human believes that insufficient context is extracted from an image, and wants to guarantee a more faithful retrieval, they can also write the caption themselves. Such an intervention into the retrieval process is not possible for other trained systems, and is simply a bonus point on top of the modular and interpretable CIR framework.

We thank the reviewer for their thoughtful comments. We appreciate that they found our paper technically sound, widely applicable, easy to follow and with rigorous experimentation. We address the concerns below:

1. Novelty of the proposed approach:
We thank the reviewer for pointing out the additional reference. However, the paper is contemporary to our submission (i.e. it was published on arxiv after the ICLR deadline), thus we could not have provided a comparison with it/known about its existence. We acknowledge several related works in Section 2 of our main paper, and highlight crucial differences between these works and our approach.

Most works on compositional image retrieval aim to train/fine-tune models specific for each dataset (i.e., training separate models for CIRR, Fashion-IQ etc.). Zero-shot Compositional Image Retrieval (ZS-CIR) instead aims to train a single model that generalizes across multiple benchmarks without the use of paired (source image, text, target image) data. The only works that explicitly focus on this task (and that we compare with) are Pic2Word [Saito et al 2023] and SEARLE [Baldrati et al. 2023], which are based on a different principle, i.e. trained textual inversion. Both methods already leverage off-the-shelf, large-scale pre-trained models as part of their pipeline.

In our work, we go one step further and propose the first training-free approach for this task, that only relies on off-the-shelf pre-trained models, while allowing for easier scalability, interpretability and the possibility of interventions.

2. PALAVRA as a baseline:
We agree that PALAVRA has not been proposed for the CIR task. However, since it is conceptually very similar to Pic2Word and SEARLE, it was originally introduced as a comparison in these works. To retain direct, comprehensive comparability, we also utilize PALAVRA as a corresponding baseline.

3. Results on Fashion-IQ:
We thank the reviewer for the suggestion. We have now updated our paper with the results on the Fashion-IQ benchmark. For the detailed results, we refer to our shared comment. In summary, we find that CIReVL significantly outperforms existing methods on all three subtasks, e.g. improving average Recall@10 by nearly 6 percentage points (pp) (28.29% versus 22.89% for SEARLE) with a ViT B/32 backbone. In addition, the simple scalability allows us to hot-swap the retrieval model to achieve additional 4pp (increasing from 28.29% to 32.19%).

4. Short Discussion on extensions and applications of our approach:
The reviewer correctly points out that our method would in-principle be applicable to videos as well as other forms of retrieval contexts, such as with interleaved images and texts (e.g. visual storytelling).

While our exact setup is structured to operate and scale especially well to the compositional image retrieval task, as long as the particularities of the problem at hand can be reasoned over in the language domain while retaining a downstream compositional retrieval or alignment target, CIReVL can certainly be adapted to account for this.

We believe this to be of definite interest for future research, and will expand our discussion to incorporate the arguments above.

5. Detect LLM Failures and future steps:
Indeed, it is not easy to automatically detect failures in the captioning model or the LLM reasoning, even though our intervention examples highlight a clear need to account for these shortcomings, and the benefits of addressing them.

For instance, one strategy may be in using another LLM to verify if the generated description is plausible or not, given the initial caption and instruction. However, doing so can only tackle issues in the LLM reasoning part, and is not able to address failures in the image captioning.

As can be seen, finding an automated mechanism to detect all manners of intervention cases is a difficult problem on its own, albeit a crucial component to improve the performance of compositional image retrieval systems. It is something we would love to see tackled in future work.

With the discussion period coming to an end, we would like to thank the reviewer again for their helpful feedback, and hope that our replies, alongside strong additional experimental results and an updated draft have addressed all questions. Of course, we are happy to continue the discussion in case of any further concerns.