Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion

We thank the Reviewer for their insightful review and for their appreciation of the clarity of our presentation and the our extensive experimental evaluation. Below we respond to each specific point raised by the Reviewer.

W1: Lack. of alternative baselines, consider evaluating alternative baselines, such as using an image-captioning model to generate captions for query images, followed by image-to-image retrieval based on these captions. This could provide a comparative perspective on the effectiveness of the proposed modality inversion techniques.

We conducted additional experiments to address concerns about the lack of alternative baselines.

Image Captioning Baseline

Following the Reviewer’s suggestion, we investigated image-to-image retrieval performance using a captioning model to generate text descriptions of query images. Given a query image, we generated a caption using a pre-trained captioning model, extracted text features from the generated caption using the CLIP text encoder, and used these features to perform retrieval. We experimented with three captioning models:

• DeCap [1], which directly generates captions from CLIP image features, making it the most comparable approach since OTI also relies only on CLIP image features;
• CoCa (LAION) [2], trained on the Laion2B dataset; and
• CoCa (MSCOCO) [2], pre-trained on Laion2B and fine-tuned on MSCOCO.

The table below summarizes the image retrieval results in terms of mAP:

In all cases captioning-based retrieval falls short of the performance of the intra-modal baseline, despite their leveraging of CLIP’s image-text alignment. Furthermore, the effectiveness of captions varies with the dataset domain: captioners struggle to produce discriminative captions for datasets featuring buildings (ROxford and RParis) while achieving better results in domains like cars (Cars). Inter-modal features derived via modality inversion (OTI), on the other hand, improve image retrieval performance on all datasets.

To understand the variability in performance of different captioning models, we report captions generated by the three models for a randomly chosen image (“all_souls_000026”) from the ROxford dataset depicting the All Souls College:

1. DeCap: “a large building with a clock tower on the front.”;
2. CoCa (MSCOCO): “an old building with two towers has a clock on it.”; and
3. CoCa (LAION): “all souls college, oxford, united kingdom.”

This example shows that the first two models fail to generate sufficiently discriminative captions, while CoCa (LAION) produces a more precise description, correlating with its higher performance among the captioning models.

We hypothesize that a more advanced captioner, such as a large multimodal language model (e.g. ChatGPT-4V), could generate more precise descriptions and potentially improve retrieval performance by better leveraging cross-modal alignment. However, comparing such an approach with OTI, which does not rely on any external data, would not be fair.

(Continued in next message)

Thank you for the clarifying experiments. All my questions have been sufficiently addressed. I think this paper addresses an important problem (poor intra-model performance of CLIP), provides a novel analysis of why this happens (due to the inter-modal CLIP loss and the modality gap), and a solution (OVI, OTI features). I also believe that demonstrating how intra-modal losses help address the modality gap problem is important for advancing the field.

I maintain my rating and recommend accepting the paper for publication.

We thank the Reviewer for their thoughtful review and for their recognition of the comprehensiveness of our experimental evaluation and the strong motivations for using inter-modal CLIP representations for intra-modal similarity comparisons. Below we respond to each specific point raised by the Reviewer.

W1: The method is quite simple. OVI is similar to OTI. There is nothing new about the inversion. But applying the inversion in this particular context can be appreciated. The authors might want to tune down the claim on the methodology contribution.

Our main claim is that ours is the first work to our knowledge demonstrating the negative effects of intra-modal misalignment in pre-trained CLIP encoders when applied to intra-modal problems like image-to-image and text-to-text retrieval, and that this misalignment can be mitigated by deriving inter-modal features by exploiting the encoder of the complementary modality. This contribution is supported by a comprehensive study of intra-modal misalignment in CLIP and the extensive experiments showing that using inter-modal representations derived via modality inversion can significantly improve the performance over intra-modal baselines.

We agree that modality inversion is not the main novel contribution of our work, it is rather a tool we use in our analysis to demonstrate that inter-modal features can be derived for intra-modal tasks. OTI, as introduced in [1], was used to combine inputs from both modalities (i.e. an input text and an input image image). In our work we use it for the very different goal of mapping a single modality to its complementary one. That is, as a mapping technique from visual to textual representations (OTI), or from textual to visual representations (OVI). The novelty in our application of modality inversion lies in the way we apply it to cast intra-modal tasks into inter-modal ones.

W2: It is a surprise to see the proposed method also helps with image classification tasks. Why? Image classification is supposed to be an inter-modal task. In the open-vocabulary setting, it is often done by comparing CLIP image features with a set of CLIP text features on the class vocabulary, which is already inter-modal.

In fact, modality inversion does not help with image classification tasks. In Table 2 (right) the rows with the green check marks correspond to the zero-shot, open-vocabulary setting using the CLIP image and text encoders. The rows highlighted in blue and indicated with red Xs instead correspond to using OTI-inverted features for query images and indeed perform worse than zero-shot CLIP given that -- as the Reviewer correctly observes -- this is an inherently inter-modal task.

We performed this experiment to show that the performance improvement on intra-modal tasks stems from leveraging inter-modal alignment and not from modality inversion itself. Applying modality inversion to such an inherently inter-modal task as zero-shot image classification transforms it into an intra-modal one and thus we expect a performance decrease due to intra-modal misalignment. Indeed, OTI maps the visual features of the input image to the textual embedding space, and thus comparing them with the textual features of the prompts makes the task intra-modal. Tab. 2 (right) confirms our hypothesis. To further confirm this claim, in Appendix F we show that the same considerations apply also to the other inherently inter-modal task of image-text retrieval.

We will revise the manuscript to improve the clarity and discussion of these intra-modal results.

W3: Missing baseline: For the image-to-image retrieval task, the authors convert the query image to a corresponding textual feature using OTI, then calculate semantic similarity between this feature and the other image features (obtained from the CLIP image encoder) to get the final results. An important baseline is missing, i.e., converting all the images into OTI features and then measuring the similarity among OTI features. This can demonstrate your proposed inter-modal CLIP representations are better for intra-modal tasks.

To address this point, we conducted the suggested experiment in which we inverted all images using OTI and measured the similarity among these OTI features for image-to-image retrieval. The results are summarized in the table below:

As shown, computing similarities among OTI-inverted features (Intra-OTI) results in performance that is slightly worse than the intra-modal baseline. This indicates that intra-modal misalignment persists even when both query and gallery images are represented in the text embedding space via OTI.

(Continued in next message)

We thank the Reviewer for their thoughtful review, for their recognition that we address an intriguing problem, and for appreciating the presentation quality and clarity of our contribution. Below we respond to specific points raised.

W1: The analysis of the intra-modal misalignment problem could benefit from more quantitative insights. In Figure 1, the authors illustrate that intra-modal similarity scores can be higher for the same class than for different ones. Providing statistics to demonstrate the significance and frequency of this issue would strengthen the argument.

To provide concrete evidence of intra-modal misalignment and demonstrate that intra-modal similarity scores can sometimes be lower for the same class than for different ones, we conducted a simple experiment using CLIP ViT-B/32 and the “dogs vs cats” dataset [1] consisting of 25K images equally distributed between dog and cat classes.

We first filtered out dog images that are not inter-modally aligned, i.e. dog images that are more similar to the prompt “a photo of a cat” than to the prompt “a photo of a dog”. Then we filtered out cat images analogously. This filtering ensures a focus on samples for which inter-modal alignment is correct. On this filtered dataset, we computed the similarities between dog images (queries) and the entire dataset. Since we exclude inter-modally misaligned samples, if inter-modal alignment implies intra-modal alignment, retrieval should be perfect -- that is, all dog images should rank higher than cat images for each query. However, our results show that this is not the case. Specifically, we observe a mean Average Precision (mAP) of 81.4% and an average R-Precision of 71.5%, where R-Precision is the precision at rank R, with R being the total number of relevant items for a given query. This means that in almost 30% of cases, cats are ranked higher than other dogs for a given dog query. A similar result was observed for cats.

We are revising the manuscript in order to incorporate this analysis -- along with a histogram visualization similar to that in Fig. 2 $c$ -- into the main paper as we feel it indeed helps support our claim and intuitive motivation illustrated in Fig. 1. We thank the Reviewer for stimulating reflection on this.

W2: The experiments lack comparative analysis. While the related works section mentions existing works that explore intra-modal misalignment, it would be beneficial to compare these methods in the experimental study to validate the paper's contributions.

Existing works that explore the problem of intra-modal misalignment [2, 3] address only the zero- and few-shot image classification tasks. Such methods require the knowledge of class names to build a set of support images [2] or to generate auxiliary texts [3]. For this reason, these approaches are not applicable to image-to-image or text-to-text retrieval.

To address the Reviewer’s concern about the lack of comparative analysis we performed experiments using additional inter-modal baselines.

Inter-modal Representations via Adapters

We trained adapters to map features from their native modality to the complimentary ones (as suggested by Reviewer pj7m). To train each adapter we used the LLaVA-CC3M dataset [4], which comprises 595K image-text pairs. Adapters are trained using a cosine loss to minimize the distance between the adapter output and the corresponding complementary features. Additionally, following Patel et al. [5], we incorporated a CLIP-based contrastive loss during training. We trained two separate adapters: one for mapping image features to text features (aligned with the goal of OTI) and another for mapping text features to image features (aligned with the goal of OVI).

First for image-to-image retrieval:

And for text-to-text retrieval:

The adapter approach improves over the intra-modal baseline for both text and image retrieval tasks, which aligns with our hypothesis that leveraging inter-modal representations for intra-modal tasks enhances performance due to CLIP’s inherent inter-modal alignment. However, on average OTI and OVI outperform the adapter-based approach. This finding is noteworthy given that OTI and OVI do not require an additional training dataset. Instead, they map individual features directly to the complementary modality without relying on external resources.

(Continued in next message)

We thank the Reviewer for their thoughtful review and for recognizing the novelty of our contribution, that our analyses provide a new direction for enhancing VLM performance on single-modal tasks, and the robustness of our experimental evaluation. Below we respond to each specific point raised by the Reviewer.

W1: The contribution is incremental, as the OTI has already been introduced by existing methods.

We respectfully disagree with the assertion that our contribution is incremental. Ours is the first work to our knowledge demonstrating the negative effects of intra-modal misalignment in pre-trained CLIP encoders when applied to intra-modal problems like image and text retrieval. This contribution is supported by a comprehensive study of intra-modal misalignment in CLIP and the extensive experiments showing that using inter-modal representations derived via modality inversion can significantly improve the performance over intra-modal baselines.

Optimization-based Textual Inversion (OTI), as introduced in [1], was used to combine inputs from both modalities (i.e. an input text and an input image). In our work we use it in a different setting and for the very different goal of mapping a single modality to its complementary one. That is, as a mapping technique from visual to textual representations, as explained in L246-L255. The novelty in our application of modality inversion (both OTI and the proposed OVI) lies in the way we apply it to cast intra-modal tasks into inter-modal ones.

W2: In zero-shot classification, current methods utilize both image and text encoders concurrently. Therefore, only utilizing the image encoder compromises fairness in the experimental comparisons for this task.

In zero-shot image classification, the inter-modal baseline (i.e. the first evaluation setting described in L409-L411 whose results are reported in the white rows of the right section of Tab. 2) utilizes both the image and the text encoder by comparing the features of the input image with those of the textual prompts. When applying OTI to the input image (i.e. the second evaluation setting described in L409-L411 whose results are reported in the blue rows of the right section of Tab. 2) we compare the OTI-inverted features with the textual prompts. Since OTI employs both encoders, the fairness of the experimental setting is ensured.

We performed this experiment to show that the performance improvement that we achieve on intra-modal tasks stems from leveraging inter-modal alignment and not from the modality inversion process itself, as explained in L83-L85. Applying modality inversion to such an inherently inter-modal task as zero-shot image classification transforms it into an intra-modal one and thus we expect a performance decrease due to intra-modal misalignment (see L314-L316 and L405-L407). Indeed, OTI maps the visual features of the input image to the textual embedding space, and thus comparing them with the textual features of the prompts makes the task intra-modal. Tab. 2 (right) confirms our hypothesis, as detailed in L412-L423. To further confirm this claim, in Appendix F we show that the same considerations apply also to the other inherently inter-modal task of image-text retrieval.