Mitigate the Gap: Improving Cross-Modal Alignment in CLIP

Thank a lot for your positive comments and recognizing the strengths of our work. Hereby, we wish to provide more clarification to the raised concerns.

W1. The extension of the sharing parameters is necessary for increasing the inductive bias towards shared modality encoders, and therefore, reduce the gap. Details about this justification is provided in L146-160. Furthermore, in order to further show the effectiveness of the Intra-Modality Separation (IMSep) loss, regardless of sharing parameters, we have conducted an ablation study for this rebuttal and report the results in the main rebuttal. Our results consistently show the effectiveness of the IMSep loss even without sharing parameters. This observation supports the effectiveness of using the IMSep loss for dual-stream encoders, e.g., using CLIP-ResNet backends. We have also added this ablation to the revised paper, Tables 1-5.

W2. Thanks for the great suggestion. As mentioned in response to W1, we have performed this ablation study and provide the results in the main rebuttal and updated the paper accordingly.

W3. Regarding the retrieval performance, our hypothesis is that it is mainly due to the complications that arise from the MSCOCO and Flickr datasets. Previous work [1] shows that these dataset include coarse-grained information. In our paper, in Figure 6A, Figure 9 and Figure 10, we also show how all models retrieve samples that are semantically relevant for the queries, but since the texts are coarse-grained, they might not necessarily be the ground truth. Such characteristics of the dataset can particularly affect the evaluation results of AlignCLIP, due to the semantic regularization in IMSep, as it is dependant on the similarity of the texts. Therefore, if the texts include coarse-grained information, AlignCLIP makes the samples closer to each other based on a higher level of semantics and therefore, can retrieve semantically close samples that are not necessary labeled as the ground truth. We suspect that this phenomenon is more noticeable in the retrieval scenario in comparison to classification datasets, as in the classification tasks, classes are semantically more disjoint.

We thank you again for your feedback on our paper. We have included your suggestion for comparing the original_CLIP+IMSep only versus (original_CLIP+)SharedCLIP in our revised paper. We hope that we have addressed your other questions and concerns sufficiently well and hope that you would consider raising your scores.

References:

[1] Chen, Weijing, Linli Yao, and Qin Jin. "Rethinking benchmarks for cross-modal image-text retrieval." Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2023

I thank the authors for addressing Q1, Q2, and Q3, and I apologize for any confusion regarding Q4.

To clarify Q4: Mitigating the modality gap (i.e., reducing the centroid difference between the two modalities) is discussed in the manuscript as a means to improve performance. However, I emphasize that achieving a zero gap does not necessarily guarantee perfect pair-wise alignment. Could you elaborate on your motivation for focusing on mitigating the gap, or explain when and why you believe this could be beneficial or not?

As requested for additional suggestions, please see W2 and Additional Considerations, which remain unanswered.

We appreciate your close attention to details and providing insightful reviews.

Regarding your proposed corrections, we checked the source code again and applied the corrections in the revised version of the paper.

In the following, we provide responses to your raised questions.

Q1. To answer this question, we expanded our experiments using the ViT-S-16 backend and provide the results in here. In general, we observe the same trend of improvements, as observed with ViT-B-16, which shows that AlignCLIP is effectively reducing the gap while improving downstream tasks for at least two transformer backends.

Results of the alignment score:

Results of zero-shot image classification:

Results of image classification with distribution shifts:

We have also provided the results of retrieval experiments in the revised version of the paper Section A.6, Tables 10-13.

Thanks for the discussions. We have now updated Line 165 in the revised manuscript as follows:

Following the original CLIP, we use the [EOS] token pooling for text embeddings and the [CLS] token embedding for the image embedding.

Thanks a lot for your great feedback on our paper and acknowledging the strength of our method and extensive experimental results.

Are all competing models trained on the same dataset?

Yes, for all comparisons, the same dataset has been used. I.e., in Tables 1-6, all models have been trained using the CC12M dataset. In Table 7, all models have been trained using the CC3M dataset.

While I believe your approach is new, a clearer link to prior work would enhance its contribution.

To the best of our knowledge, our proposed semantically-regualized InfoNCE contrastive loss for intra-modality separation (IMSep) is quite novel and has not been proposed in prior work.

Thanks for taking the time to review our paper. In the following, we provide more clarifications to the raised points.

W1. We would like to emphasize that our proposed semantically-regularized IMSep loss is a novel approach in contrastive learning, allowing for controlling the harshness of the optimization of the denominator in the InfoNCE loss. To the best of knowledge, this objective function has not been proposed before and can be utilized in other scenarios where such regularization in the InfoNCE loss is required.

W2-W4. Thanks for your input. We implemented the naive approaches introduced in [1, 2] and updated our SOTA comparison, respectively. The updated results are as follows:

We observe that AlignCLIP achieves the best results in almost all the tasks. However, in classification with distribution shift for ImageNet-A, we notice that naive shifting approach remarkably improves the classification accuracy. We attempted at getting some insights from the original papers $1, 2$ and notice that the distribution shift experiments have not been studied in neither $1, 2$ . We have included this comparison in the revised version of our paper as well.

W5. We respectfully ague that L198-202 are indeed correct. In L198-200, we describe the situation where the samples are semantically close (L199), and in L200-202, we describe the opposite scenario, where the samples are semantically dissimilar (L201).

W6. Thanks for pointing out the typo in L269. We have corrected this in the revised version of the paper.

Thanks for sharing your feedback and raising your concerns. In the following, we provide individual responses to the weaknesses you have mentioned.

W1. Regarding the technical novelty, please note that AlignCLIP’s contribution is twofold: 1. Sharing the transformers, which we have acknowledged in the paper that has been proposed by previous work, and we only extend it to include all parameters that can be shared, in order to increase the inductive bias towards shared cross-modal encoders, 2. A novel semantic-regularized intra-modality separation InfoNCE loss function proposed in Section 3.2, i.e., IMSep. In our rebuttal, we have provided the ablation of using IMSep loss without sharing parameters, in order to show the effectiveness of IMSep, regardless of the sharing architecture. The results are provided in the main rebuttal and in the revised version of the paper, Tables 1-5. This ablation study shows that our novel IMSep objective function is effective in improving all of the downstream performances while reducing the modality gap regardless of sharing the transformers.

W2. In order to test this, we performed the ablation of using IMSep on the dual encoder scenario, i.e., without sharing the transformer encoder, and reported the results in the main rebuttal and revised Tables 1-5, accordingly. We observed the improvement of the gap while enhancing all downstream performances when using the two-tower architecture.

W3. Thanks for the suggestion. We have already provided this analysis in our submitted paper. As mentioned in Section 1 of our paper, the work you mentioned ( $3$ ) has used naive shifting for closing the gap which can damage the cross-modal distances. In Section A.1 of our submitted paper, we have provided a theoretical argument of how this shifting can change the relative cross-modal distances. As a result, authors in $3$ could not achieve a consistent outcome from reducing the gap. However, in AlignCLIP, we reduce the gap by enforcing an inductive bias towards shared modality encoders as well as training a semantic separation of the unimodal embeddings, which results in a more meaningful structure of the embedding space, as it considers the semantic relationships.

We thank you again for your reviewing our paper and providing feedback. We hope that we have addressed them sufficiently well. We would be happy to provide more clarifications and appreciate it if you consider raising your scores.

Thank you for the insightful comments and close attention to details. We very much appreciate the raised points.

W.1: Potentially limited applicability

Thank you for the comment. As you suggested, we performed a new ablation study for the case of using IMSep loss without sharing parameters, which reduces the dependency of our method on the shared transformer architecture. We have provided the results in the main rebuttal and show that using the IMSep loss noticeably improves the alignment as well as the downstream performances.

W2: Comparison to baselines

Thanks for your valuable input. We implemented your first suggestion and trained CLIP and CyCLIP models ourselves, with the same hyperparameters as our model’s and report the results:

Our new comparisons show that the AlignCLIP model achieves even stronger results in comparison to SOTA models.

As for the DFSep model, since the authors have not published their source code, given the limited time frame, we had to carefully prioritize our tasks. Nevertheless, we employed your second suggestion for this model, and have added more description in the caption of Table 7 to be more fair.

In response to your questions:

Q1. The usage of this prompt is encouraged by the original CLIP paper as stated in the supplementary material of $1$ , Section H.1:

For both these datasets we prepend the prompt “a photo of” to the description of each image which we found boosts CLIP’s zero-shot R@1 performance between 1 and 2 points.

Therefore, we applied the same prompt template and also observed a noticeable boost in the zero-shot performances.

Q2. Thanks for your valuable suggestion. We expanded our experiments to add this ablation study and reported the results in the main rebuttal. We have also included these result in the revised version of the paper, Tables 1-5.

Q3. Thanks for suggesting this valuable analysis. Yes, we have tracked the temperature values during the training . As mentioned in Section A.2 in the paper, the initial temperature value was set to 0.07 for all models. The final learned temperature value for the original CLIP is 0.015, for sharedCLIP is 0.011, and for AlignCLIP is 0.010. This concludes that AlignCLIP, while having the least temperature value at the last epoch, achieves the least modality gap, showing the effectiveness of our proposed refinements.

We have tracked the temperature value at each step by logging the logit scale which is the inverse of the temperature value, i.e., 1/temperature. Furthermore, we have provided the plots for each model in Figure 11 in Section A.5 in the revised version of the paper.

We thank you again for your insightful comments and questions. We have included all of them in our revised paper and wish that we have addressed them sufficiently well. We would be happy to provide more clarifications and appreciate it if you consider raising your scores.

References: [1] Radford et al., Learning Transferable Visual Models From Natural Language Supervision (http://proceedings.mlr.press/v139/radford21a)

W2. We have now updated the manuscript and added further clarifications in L507-514 of the revised manuscript as follows:

For a fair comparison with CLIP andCyCLIP, we re-train these models with the same hyperparameters as SharedCLIP and AlignCLIP. Furthermore, we train versions of SharedCLIP and AlignCLIP with the CC3M dataset and compare the resulting alignment scores as well as the downstream performance in Table7. As for the DFSep model, we report the numbers directly from the paper [Jiangetal.(2023)]. This model uses a ResNet50 backend for the vision encoder and the BERT model for encoding text. In contrast to the rest of the models compared in Table7, DFSep is trained with the batch size of 128, embedding dimensionality of 1024, learningrate of 5e-4 and warmup value of 10000.

Thanks for your discussions and for considering to increase the score.

Thanks a lot for reviewing our paper, recognizing the strengths and providing valuable feedback.

W1 and W2: Evidence of modality gap

Thanks for the feedback. Since the problem of the modality gap in CLIP has already been extensively introduced in previous work [1, 2], we have provided the corresponding citations in Section 1 and presented the DOSNES visualization, and dedicated the rest of the paper to proposing solutions for mitigating the gap.

W3: Learning each modality

Thanks for sharing your concerns. However, we sincerely draw your attention to our extensive experimental results on zero-shot image classification, classification with linear probing, classification with natural distribution shift, zero-shot multi-modal retrieval and fine-tuned multi-modal retrieval which consistently show that SharedCLIP and AlignCLIP achieve superior performance, therefore, showing that the image and text features are indeed being learned better.

W4. Considering text modality in the objective function.

We have already included this in our submitted paper and provided the results of using the text-text separation in the case of intra-modality loss function in Table 8 and 9 of the paper. Our experiments show that the addition of text modality is indeed not helpful and enforcing the separation on the image embeddings outperforms other possibilities.

For more convenience, we report the comparisons in zero-shot image classification here as well. TT stands for text-text separation, II represents image-image separation and II-TT is the case where both text-text and image-image separation have been used.

Additionally, we provide the ablation for the zero-shot multi-modal retrieval experiments as well. In the following, the performance of image-to-text (I2T) as well as the text-to-image (T2I) retrieval evaluated by R@{1, 5, 10} is reported. As can be seen, the addition of the IMSep loss, without sharing the transformer encoders, noticeably outperforms the CLIP model, suggesting the effectiveness of the proposed IMSeploss function regardless of sharing architectures:

Lastly, the fine-tuned multi-modal retrieval experiments are provided in the following table:

This ablation study concludes that the addition of the IMSep loss to the clip loss is indeed effective in improving the downstream performances.

We present the results of our ablation study, requested by $`WiMx, J1QT, ZbcG, mh82`$ in this response. In all tables, “IMSep” refers to the case of using the addition of IMSep loss without sharing the transformer between the two modalities.

Starting with the alignment scores, it is observed that IMSep without sharing parameters improves the alignment score in comparison to CLIP:

Furthermore, the results of the zero-shot image classification is as follows. It is observed that IMSep loss without sharing the parameters achieves an improvement in the results when compared to the original CLIP model:

For image classification with linear probing, the following table shows that the IMSep loss without sharing the transformers substantially improves the results when compared to the original CLIP model:

Moreover, we provide the results of image classification with distribution shifts. These results also show that the addition of IMSep improves the accuracy in comparison to the CLIP model: