TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives

We are encouraged by your review! We thank you for your comprehensive evaluation of our paper. We are grateful that you found TripletData alone to be of significant value and that our experiments are comprehensive.

Please find the requested clarifications below.

Response to Weaknesses

[W1] Limited Scaling: We acknowledge that scaling our experiments further in academic settings is challenging. However, to approximate this, we experimented with increasing the data (i.e., WordNet synsets), as shown in Figure 3. Our findings indicate a consistent upward trend, with TripletCLIP outperforming the baseline CLIP in terms of compositionality while maintaining zero-shot classification performance.

[W2] Selection of LLMs and T2I Models: Grounding the selection procedure of LLM to generate negative captions is indeed essential. We performed additional experiments with three LLMs and trained NegCLIP++ to evaluate their behaviors. The results show that Phi-3 is the best overall. However, at the time of submission, Phi-3 wasn’t released, so we selected the current second-best option, Mistral. The selection of T2I models is straightforward, as their goal is to generate images that truthfully align with the text. Therefore, selecting the best open-source T2I model is a logical choice.

[W3] We will update the equations in the final draft (if accepted). Thank you!

Response to Questions

[Q1] Evaluations on valdiation subset of TripletData: Thank you for suggesting this interesting analysis. We evaluated the CC12M pre-trained models on a 50,000 random subset of the CC3M dataset and reported the scores below. As expected, TripletCLIP significantly boosts the performance of the baselines. However, we also partially attribute this to the spurious correlation learned from the data (related to Q3). At the same time, we note that models are not fully converged, so there is very little chance of overfitting on this spurious correlation.

[Q2] Clarification on TripletData size: We created TripletData on top of LaCLIP, which provides a ~2.9M subset of captions for CC3M and a ~10M subset for CC12M, totaling ~12.9M. However, we observed failures in downloading several images during training, leading to a further decrease in image-text pairs during the training phase. We will clarify this in the final draft (if accepted).

[Q3] Potential real vs. synthetic distribution mismatch: Yes, a distribution mismatch exists between real and synthetic images. The generated images are of higher quality than CC3M/12M, potentially adding spurious correlations (partially observed in the Q1 table). However, other generated images represent similar objects with different semantics/relations that are not present in the real data. Despite TripletCLIP learning these biases, it still improves compositionality.

[Q4] Winoground clarification: Winoground data contains only 400 instances, which may not be statistically significant for evaluating models trained with lower resources. Even with finetuning on top of the pretrained models, we observed a decrease in performance. Ideally, TripletData should replace the Winoground. We will plan to release the high-quality subset of TripletData as an evaluation set for the community to rely on. Thank you for inspiring this idea.

[Q5] Yes, this is a typo. We meant TripletData. Thank you for pointing this out.

We hope this clarifies all the remaining questions. Additionally, we offer a summary of responses to other reviews for your reference in the global response.

We appreciate your review of TripletCLIP and the time you've dedicated to it. Thank you for the detailed and holistic review of our work. We are delighted that you found it valuable to the research community.

Comparison with Related Works [W1]

Firstly, we want to clarify the key baselines selected in the main paper, reflecting the current state of research at the pretraining stage:

• NegCLIP: SOTA baseline for negative captions based on the latest CVPR’24 work [12].
• NegCLIP++: LLM-enhanced negative caption similar to [6,7].
• CLIP + HN: Real-world hard negatives filtered from existing datasets.

We further divided the related works into two categories:

Orthogonal Works [1,6-9]

Our contributions are orthogonal, meaning our findings can be combined with theirs for further improvements. For instance, CyCLIP [1] introduces a novel regularizer but does not include hard negatives. [6,7] introduce synthetic captions similar to NegCLIP++. [8] focuses on detailed caption-based NegCLIP fine-tuning. [9] introduces scene graph-based negative captions.

We retrained CLIP and CyCLIP models with and without TripletLoss using the CC3M training dataset and a batch size of 512. Adding TripletLoss consistently improved performance over baseline CLIP and CyCLIP:

Negative Captions and Images [2-5]

These works are closer to TripletCLIP. Specifically, [2] utilizes the video dataset to perform hard negative mining and [3] focuses on object-centric caption and image editing for hard negatives, while we focus on free-form captions and image editing that modifies semantics globally. [4] introduces a synthetic data pipeline for benchmarking and proposes a simple extension to NegCLIP pretraining, aligning with our analysis in Table 3, which shows TripletCLIP’s significance. [5] focuses on a "negation" and utilizes a method similar to [4].

Although we could incorporate these negative captions [6-9] along with the TripletCLIP procedure for further improvements, due to time constraints, we perform direct comparisons with finetuning-based experiments and leave the joint optimization as future work.

We will include this detailed comparisons in the related work to make it more comprehensive in the final draft.

Finetuning Based Experiments [W2 + W3]

Our paper focuses on pre-training, as fine-tuning focuses on post-training solutions, which may not be generalizable. That said, we performed additional finetuning experiments with hyper-parameters similar to [3,5,7] (w/o LoRA) and compared against various baselines [3,5,7,8], whose public checkpoints are available. Our results show that TripletCLIP improves compositionality and outperforms the almost all baselines. Additionally, the drop in retrieval and zero-shot classification is attributed to the vision encoder (Table 7), indicating the limitations of existing pre-trained vision encoders to represent semantics. This another reason to conduct the pretraining based experiments for comprehensive evaluations.

Here, DAC [8] introduces detailed human-annotated captions with additional loss functions to incorporate multiple sentences of the captions, which is crucial this performance. Our work focuses on synthetic data with short captions. Future work could explore combining both approaches together.

Additional Clarifications on W2

Lines 70-72 convey that CLIP models are key components behind VLMs and T2I models. Our method can potentially improve CLIP models, directly impacting downstream tasks. TripletData could inspire future improvements (noted by ZTUH), possibly requiring different pretraining like DPO. We leave this as future work. However, we will make it clear in the camera-ready draft.

Additional Benchmark Evaluations

We have already provided additional comparisons on three widely adopted benchmarks in Table 9. Importantly, the newly suggested benchmarks [2,4,10,11] are not well-adopted by the community. But below are the additional evaluations on COCO-counterfactuals [10] that is very similar to [2]. Interestingly, all models perform consistently with no significant improvements over the baseline, possibly due to the evaluation data's difficulty or noise. Having said that, due to the time constraints associated with rebuttal phase, we could not extend large-scale evaluations on another two benchmarks.

Response to Remaining Minor Questions

• [Q1] We will add the suggested citations in the final draft (if accepted).
• [Q2] We will clarify the section we are referring to, specifically Figure 2 (main paper) and Figure 6 (appendix).
• [Q3] By “real” hard negatives, we mean the hard negative image-text mining within the training datasets instead of synthesizing them.

We trust that our response adequately addresses your concerns and encourages you to reevaluate our submission. We look forward to the discussion.

---

[12] Zhang et. al., “Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding,” CVPR 2024.

Thank you for the detailed response. I have no remaining concerns and will adjust the score to 6.

I believe that including additional experiments with CyCLIP and fine-tuning comparisons will provide valuable references for future research. It would be great if that part of the codebase were released.

Thank you for your review. We respectfully disagree with the claims regarding our work's limited novelty and contributions. Other reviewers have unanimously recognized numerous strengths in our work despite its straightforward nature. Therefore, we would like to reiterate our research's key strengths and contributions.

Novelty and Contributions

Importance: CLIP models have demonstrated limited performance in terms of compositionality and tend to behave as bags-of-words, as observed by [1]. This work suggested introducing negative mining, particularly negative captions, could enhance model performance. Following this, several studies have focused on improving synthetic hard negative captions [2,3,4,5], while others have aimed at enhancing data quality [6,7,8]. However, with recent advances in image generative models, it remains unclear whether such tools can improve compositionality. Another recent study introduced training CLIP on a fully synthetic dataset [9], but it required three times more data to achieve performance comparable to real data.

Novelty/Contributions: To address these limitations and improve image-text understanding, we propose TripletCLIP. TripletCLIP leverages LLMs to generate hard negative captions (similar to [3,4]) and introduces hard negative images focusing on various image semantics. Additionally, we introduce a novel contrastive pretraining loss, TripletLoss, which enhances the usability of our synthetic data generation pipeline. To be specific, our contributions are as follows:

• TripletCLIP significantly improves baseline performance by an absolute 6-10% across benchmarks and shows absolute 4-7% improvements over the baseline proposed by [1].
• We have released approximately 13M synthetic image-text pairs, complementing real-world datasets like CC3M and CC12M.
• Additional experiments (Table 7) indicate that vision encoders are the primary source of compositionality limitations in pretrained CLIP models, which were previously unknown. TripletCLIP offers a promising solution to overcome this through careful training on hard negative images, specifically using TripletLoss.
• Experiments with increasing concept diversity (Figure 3) further validate our approach, demonstrating that TripletCLIP consistently improves performance with larger dataset sizes.

We trust that our response adequately addresses your concerns regarding novelty and encourages you to reevaluate our submission. We also kindly request you to consider the feedback from other reviewers and our responses, contributing to a comprehensive assessment of our work.

---

[1] When and why vision-language models behave like bags-of-words, and what to do about it? ICLR 2023.

[2] Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Fine-grained Understanding, CVPR 2024.

[3] Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models, NeurIPS 2023.

[4] Teaching Structured Vision&Language Concepts to Vision&Language Models, CVPR 2023.

[5] Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality, EMNLP 2023.

[6] DataComp: In search of the next generation of multimodal datasets, NeurIPS 2023.

[7] Demystifying CLIP Data, ICLR 2024.

[8] DreamLIP: Language-Image Pre-training with Long Captions, ArXiv 2024.

[9] SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?, ArXiv 2024.

Thank you for your time and consideration given to our paper. We are delighted that our work is recognized as well-written, easily reproducible, and effective. We appreciate your belief in our approach and its potential to inspire similar strategies in downstream tasks.

In response to your inquiries, please find our clarifications below:

Ablation on Generative Model Choices

While we acknowledge that exploring multiple LLMs and T2I models could be beneficial, our focus in the paper was shaped by three primary considerations:

• The semantic manipulation of text by an LLM is paramount, regardless of the specific LLM used. For better comprehensiveness, we have performed an additional ablation study, which is detailed below.
• T2I models must precisely synthesize images from hard negative captions. Hence, we selected SDXL-Turbo, a state-of-the-art and efficient open-source model, thus not necessitating further ablation on T2I model choices.
• Synthesizing negative captions and images at scale demands significant resources, making holistic evaluations challenging.

For better comprehensiveness, we conducted experiments with different LLMs to generate negative captions, which is a crucial component. The table below presents results from using three LLMs to generate 3M negative captions each for the CC3M dataset. We then trained NegCLIP++ models to assess the effectiveness of these synthetic captions. Notably, Gemma-2 significantly reduces compositionality, while Phi-3 performs best overall. The Phi-3 model was released after the NeurIPS deadline; hence, we use Mistral (second-best choice) in this case.

Overfitting Due to Synthetic Data

We agree that synthetic data can lead to overfitting. As noted in our paper and supported by recent work on fully synthetic CLIP models like SynthCLIP [1], three times more synthetic data is required to match the performance of real data due to limited diversity. Our research was motivated by the question: “What kind of synthetic data can enhance performance?” We found that generating hard negative captions and images is the optimal solution. Preliminary analyses on data scaling trends, shown in Figure 3, demonstrate consistent performance improvements with our approach as we scale the synsets. While our experiments did not observe overfitting (Figure 3), it remains a potential issue if scaled to billions of hard negative image-text pairs. This topic extends beyond the scope of our current work, and we suggest further exploration by the community in future research.

We hope these clarifications address all your questions and enhance the comprehensiveness of our work. Additionally, we provide a summary of responses to other reviews in our global response for your reference.

---

[1] Hammoud, Hasan Abed Al Kader, Hani Itani, Fabio Pizzati, Philip Torr, Adel Bibi, and Bernard Ghanem. "SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?." arXiv preprint arXiv:2402.01832 (2024).