Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP

Thank you for your helpful comments!

Q1: More relevant experimental results are expected, such as results from a wider range of downstream tasks.

A1: Thank you for your suggestion. To add more experiments on downstream tasks, we have added experiments on retrieval accordingly and summarized the zero-shot results in table below, including Flickr-30k (1k) and MSCOCO (5k) following the CyCLIP paper [1]. As also noted in [1], text retrieval per image is easier than image retrieval per caption, due to the nature of both datasets containing 5 captions for each image. The trend of our results align with that reported in [1]. Lastly, while CyCLIP does not provide consistent improvement over CLIP (in our experiment setting as well as in their reported results), our regularization term provides consistent improvement to both CLIP objective and CyCLIP objective. This has also been updated in our Appendix C.1.

Flickr-30k (1k)

MSCOCO (5k)

---

Q2: The article exclusively analyzes and experiments with CLIP, without thoroughly exploring the applicability of this new methods to other relevant contrastive learning approaches.

A2: Since we are the first paper that theoretically studies the transferable representation learning of CLIP, the primary focus of our paper was an in-depth analysis of the CLIP model. Due to the space limit, we only consider the vanilla CLIP model in our theory and, additionally, the CyCLIP [1] model in the experiment. We acknowledge the importance of exploring other contrastive learning approaches and consider this an important direction for future research.

---

Q3: Can one simply replace the CLIP component in all CLIP-related work with CLIP+Reg to achieve a better performance?

A3: We think so. Replacing the CLIP component with CLIP+Reg in various CLIP-related projects is feasible and can potentially lead to improved performance. In principle, our regularization is quite simple and can improve the alignment between image-text pairs, as demonstrated by the above experiment results. However, achieving a successful replacement might necessitate additional effort in exploring various related works to fully leverage the benefits of our regularization.

---

[1] Goel et al. "Cyclip: Cyclic contrastive language-image pretraining." Advances in Neural Information Processing Systems 35 (2022): 6704-6719.

Q3: Some notations and definitions in the paper can be challenging to follow. ('one-to-one mapping' or 'one-to-one matching')

A3: Thank you for the valuable suggestion, we have unified the terminology as one-to-one mapping.

---

Q4: Expanding the range of experiments to include various downstream tasks, rather than solely focusing on zero-shot and Linear probing…

A4: Our research concentrates on the zero-shot transfer capabilities in the CLIP. This focus aligns with the core tasks (zero-shot and linear probing) emphasized in the original CLIP paper and its subsequent developments, such as FILIP [1] and CyCLIP [4]. To add more experiments on downstream tasks, we have added experiments on retrieval accordingly and summarized the zero-shot results in table below, including Flickr-30k (1k) and MSCOCO (5k) following [4]. As also noted in [4], text retrieval per image is easier than image retrieval per caption, due to the nature of both datasets containing 5 captions for each image. The trend of our results align with that reported in [4]. Lastly, while CyCLIP does not provide consistent improvement over CLIP (in our experiment setting as well as in their reported results), our regularization term provides consistent improvement to both CLIP objective and CyCLIP objective. This has also been updated in our Appendix C.1.

Flickr-30k (1k)

MSCOCO (5k)

---

[1] Yao, Lewei, et al. "FILIP: Fine-grained Interactive Language-Image Pretraining." International Conference on Learning Representations. 2021.

[2] Li et al. "Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pretraining Paradigm." International Conference on Learning Representations. 2021.

[3] Mu et al. "Slip: Self-supervision meets language-image pre-training." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[4] Goel et al. "Cyclip: Cyclic contrastive language-image pretraining." Advances in Neural Information Processing Systems 35 (2022): 6704-6719.

Thank you for your helpful and detailed comments!

Q1: The paper's primary focus appears to be on CLIP's zero-shot transferability. While this is undoubtedly a significant aspect, it's worth considering that CLIP's robust zero-shot performance results from a strong semantic space based on extensive vision-semantic data. Therefore, an exploration of the visual-semantic alignment aspect could be an intriguing avenue for further investigation.

A1: Thank you for your valuable suggestion. While our primary focus is CLIP’s zero-shot transferability, we indeed considered the visual-semantic alignment in our analysis. As noted in Remark 4.3, image-text pairs sharing the same topic tend to score higher, indicating a visual-semantic alignment in CLIP's learned representations. Furthermore, the margin discussed in our paper can be considered a useful metric for studying this alignment. We've expanded on this in Appendix A for further clarification.

---

Q2: In introduction section, the author cites "blue sky" and "white cloud" as examples of unique features. However, these instances might be seen as special cases. As CLIP is based on a large amount of vision-semantic data, it's possible that the missing elements could appear in various other captions. Therefore, I question the significance of this problem.

A2: In the introduction section, we presented an example to illustrate the existence of such an issue. While CLIP is trained on extensive image-caption datasets, each image is paired with only a single, brief caption. It is crucial to note that CLIP's pre-training data are web-sourced image-caption pairs, characterized by their succinct and often non-elaborate nature, as these captions are not meticulously crafted by annotators. Consequently, instances where captions do not comprehensively describe the corresponding image are common in such datasets.

While the ground truth object list is not available for CC3M, we can first consider the MSCOCO dataset that has a ground truth object list. Analyzing the 410,600 image-caption pairs from its training data, we identified 330,843 pairs wherein the caption failed to include at least one object based on exact matching criteria. This amounts to 80.58% of the dataset. We have included the distribution bar plot in Appendix C.3.

We also provide several examples of image captions, along with the ground truth objects in the image. We highlight the objects that are missed from the caption.

• Caption: “A restaurant has modern wooden tables and chairs.”

• Objects: [potted plant, dining table, book, vase, chair]

• Caption: “A bicycle parked in a kitchen with a stove and cabinets.”

• Objects: [bicycle, bottle, cup, toaster, sink, spoon, bowl, oven]

• Caption: “A person is cutting a roast with a fork and knife.”

• Objects: [person, oven, knife]

As shown above, humans do not extraneously identify all conceivable objects in an image when trying to describe it, let alone the background information such as the scene. If the reviewer browses through some examples of MSCOCO and CC3M, it will be a frequent case that the captions cannot fully depict the image details. In Appendix C.3, we further provide several image-caption examples from CC3M.

Dear Reviewer CWhP,

Thank you again for your insightful comments. We are following up to engage in further discussion and address any outstanding questions you might still have. If our rebuttal and the revisions to our paper have satisfactorily addressed your concerns, we respectfully hope that you could reconsider your assessment of our work.

[1] Yao, Lewei, et al. "FILIP: Fine-grained Interactive Language-Image Pretraining." International Conference on Learning Representations. 2021.

[2] Li et al. "Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pretraining Paradigm." International Conference on Learning Representations. 2021.

[3] Mu et al. "Slip: Self-supervision meets language-image pre-training." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[4] Goel et al. "Cyclip: Cyclic contrastive language-image pretraining." Advances in Neural Information Processing Systems 35 (2022): 6704-6719.

[5] Xiaohua Zhai et al. "Lit: Zero-shot transfer with locked-image text tuning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[6] Jean-Baptiste Alayrac et al. "Flamingo: a visual language model for few-shot learning." Advances in Neural Information Processing Systems 35 (2022): 23716-23736.

Thank you for your strong support and valuable feedback！We answer your major comments and questions as follows.

Q1: Current pre-training experiments are all based on the CC3M… it is recommended to conduct experiments on datasets with different sizes.

A1: Thank you for your valuable suggestion. CLIP serves as a strategy for learning features that can be transferred between text and images. Unfortunately, the dataset used by Open AI is not open-source. Therefore we utilize the CC3M dataset which is a standard and widely used dataset used in prior research on CLIP [1-4].

---

Q2: In Table 1, why incorporating the regularization term into the contrastive objective is harmful to DTD?

A2: Thank you for raising this point. While our method can improve on 5 out of 6 downstream tasks, Describable Textures Dataset (DTD) presents a special case where the regularization term can get a worse zero shot performance. We conjecture that the unique properties of texture data in DTD require a more fine-grained approach of regularization. Therefore, we conduct more experiments on the DTD dataset and find that the combination of our regularization and CyCLIP can improve the original CLIP on the DTD data set.

---

Q3: It seems that the results of CyCLIP in Table 1 and Table 2 are inconsistent with the original CyCLIP paper.

A3: Indeed, our results for CyCLIP are better than the results reported in the original CyCLIP paper on most tasks. This is due to the different experiment setting of our work, as we train from pre-trained image and text encoders while they trained from scratch. Similar settings can be found in LiT [5] and Flamingo [6] that train from pre-trained single-modality models.

---

Q4: Can the proposed regularization technology work in CyCLIP?

A4: Yes, it indeed can be used for CyCLIP and leads to improvements on most tasks. We have added the results for CyCLIP+Reg as follows.

Zero-shot

Linear probing

---

Q5: Is the method equally effective for other downstream tasks such as retrieval?

A5: Thank you for pointing us to the retrieval task. We have added experiments on retrieval accordingly and summarized the zero-shot results in the table below, including Flickr-30k (1k) and MSCOCO (5k) following the original CyCLIP paper [4]. As also noted in [4], text retrieval per image is easier than image retrieval per caption, due to the nature of both datasets containing 5 captions for each image. The trend of our results align with that reported in [4]. Lastly, while CyCLIP does not provide consistent improvement over CLIP (in our experiment as well as in their reported results), our regularization term provides consistent improvement to both CLIP objective and CyCLIP objective. This has also been updated in our Appendix C.1.

Flickr-30k (1k)

MSCOCO (5k)

Q4: The experiments are conducted on a relatively small dataset compared to CLIP.

A4: Unfortunately, the dataset used by Open AI is not open-source. Therefore we utilize the CC3M dataset which is a standard and widely used dataset in prior research on CLIP [1-4]. This dataset is also used in UniCL as the reviewer mentioned, referred to as GCC3M. Many of their experiments (ImageNet-1K, GCC3M, GCC3M+ImageNet-1K) are done at smaller or the same scale.

---

Q5: The derivations make sense but they are under several assumptions.

A5: In order to theoretically analyze CLIP, we have to make certain assumptions, like all theory papers do. The assumptions streamline the analysis, rendering the problem more amenable. More importantly, the assumptions made in our paper are quite standard and reasonable. In detail, in Assumption 3.1, we make a conditionally independent assumption which is commonly used in the theoretical literature of self-supervised learning (as we mentioned in Remark 3.2). In Assumption 4.1, we make a completeness assumption to allow the existence of a good representation of image-text pairs. We acknowledge that the relaxation of our assumptions would be a interesting future direction.

[1] Yao, Lewei, et al. "FILIP: Fine-grained Interactive Language-Image Pretraining." International Conference on Learning Representations. 2021.

[2] Li et al. "Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pretraining Paradigm." International Conference on Learning Representations. 2021.

[3] Mu et al. "Slip: Self-supervision meets language-image pre-training." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[4] Goel et al. "Cyclip: Cyclic contrastive language-image pretraining." Advances in Neural Information Processing Systems 35 (2022): 6704-6719.

Thank you for your constructive comments!

Q1: Contrastive learning has been widely studied in the community. Also, how does the proposed solution compare to those methods?

A1: Thank you for pointing out these related works. We have added discussions on those papers. Similar to CLIP and ALIGN, paper (a) proposed a contrastive learning scheme UniCL for multi-modal learning. Unlike CLIP that primarily rely on image-text data, UniCL uniquely integrates image-label data (ImageNet-1K). This integration enables them to group data with identical labels, allowing for a broader range of positive pair identifications within the dataset. There is no regularization term proposed in this paper; rather, it’s a different contrastive learning method. We have included this paper in discussion as an improvement over CLIP, alongside other methods we discussed such as SLIP, FILIP, and CyCLIP. Moreover, for the paper that actually proposed regularization terms for CLIP (CyCLIP), we indeed made a thorough discussion.

The (b,c) focuses on unimodal contrastive learning, and thus cannot adequately explain the zero-shot transfer capability of CLIP. As we emphasized in our paper, we focus on multi-modal contrastive learning, which has not been studied from a theoretical perspective and has its special challenges resulting from multi-modality. Ours is the first paper to provide a formal understanding of transferable representation learning and zero-shot transfer in CLIP. We have added (b,c) to the related work section and as well as the detailed discussion in Appendix A.

---

Q2: The authors propose only to regularize the distance between positive pairs, but there is no ablation comparison to the variants that regularize both or negative pairs only.

A2: In our paper, we did clarify that accurately identifying truly negative pairs in image-text data is challenging, as different pairs within the same batch may share common features. Since web-scale image-text data are inherently unlabeled, it is different from the unimodal case where negative pairs can be easily determined by the class labels. As shown in Figure 1, the image-text pairs containing similar features could appear in the same training batch and be mistakenly considered as negative pairs. Consequently, our proposed regularization term only incorporates positive pairs.

We have added an ablation study by taking all off-diagonal pairs as negative pairs and adding them as a regularization term. The table presented below compares the effects of regularizing positive pairs (Pos) against off-diagonal pairs (Neg) in terms of zero-shot accuracy. While the regularization on positive pairs markedly improves performance, regularization on off-diagonal pairs yields only marginal enhancements on some datasets and no improvement on others. This unstable performance may be attributed to the presence of positive pairs among the off-diagonal elements in the unlabeled image-text data. We have added a discussion on this in Appendix C.2.

---

Q3: The introduction states that the contrastive learning objective in CLIP may not cover all the positive pairs, which makes sense. However, it is unclear how the proposed solution addresses this issue.

A3: In our paper, we acknowledge the challenge posed by the fact that "Contrastive learning objectives in CLIP may not cover all positive pairs," both from theoretical and empirical standpoints. Our primary goal is to rigorously explore transferable representation learning in CLIP, and to introduce a theoretically-guided regularization that enhances the original CLIP, particularly focusing on the concept of “margin”. Therefore, our approach primarily addresses this issue from a theoretical angle by incorporating this mismatch into our analysis. We recognize the significance of empirically addressing this issue, but it goes beyond the scope of our current work. The literature [a] you mentioned is a milestone work in this area, which introduces a new loss function. Analyzing why this improved loss [a] outperforms CLIP from a theoretical perspective can be an interesting future direction.

Dear Reviewer 7qr5,

Thank you for taking the time to review our paper and providing valuable feedback. We would like to follow up with you and provide additional experiment results regarding Q2 on regularization of negative pairs. In our previous rebuttal, we have explored regularizing the negative (off-diagonal) pairs as compared to positive pairs. Here, we have provided additional experiment results on regularizing both positive and negative (off-diagonal) pairs, which are summarized in the table below. These results demonstrate that regularization of both pairs leads to improvements over only regularizing negative (off-diagonal) pairs across most datasets. However, it still underperforms our proposed method CLIP+Pos, which only regularizes positive pairs. This experiment further supports our initial observation that some off-diagonal image-text pairs may share common features, making their categorization as negative pairs potentially detrimental. Identifying true negative pairs in unlabeled image-text data could be an interesting direction for future research.

We look forward to your further feedback and are more than happy to discuss any remaining questions you may have. If our rebuttal and the revised paper have effectively addressed your questions and concerns, we sincerely hope you could reconsider your evaluation of our paper. Thank you!