Data De-Duplication and Semantic Enhancement for Contrastive Language-Image Pre-training

Thank you for your insightful advice and valuable questions, we will respond to your concerns point by point.

Q1: The key intuition and motivation of the proposed method. Explanation of the contradiction between the effectiveness of ($\text{D}^3$) and the other three modules e.g. text augmentation, DCTM, and MSTM.

The key intuition and motivation of the proposed method is that construct a semantically diverse multimodal dataset and present an efficient training algorithm for better representation learning.

The generated texts are completed by one-time processing, which can be reused to train other models, the time does not require double-counting. The efficient training algorithm is proposed to utilize semantically enriched datasets. We also provide the specific calculation time for each module.

Specifically, as Q4 of review Fx4U, we counted the time for all modules. It takes about 175 hours to rewrite one text from LLaMA for the entire Laion400M dataset on 32 A100 GPU machines, and it takes only 75 hours to generate texts from LLaVA with batch inference. Training baseline CLIP on LAION-400M with large ViT-L/14 almost takes 550 hours on 32 A100 GPU machine. Training with half of the clustered augmented data can achieve comparable performance but its training time is reduced by half. Clustering time is negligible as compared to text generating, it only takes a few hours on one A100 GPU machine. The total training cost has decreased a little overall (550 vs 525) on 32 A100 GPU machines.

Q2: The limination of novelty.

Please refer to General Response.

Q3: The effect of the pre-trained model e.g. DINO in data de-duplication.

Tab.7: Ablation study for clustering pre-trained model on ImageNet(IN) datasets for Zero-shot image classification (%), all experiments are used 50% clustered data.

As per your suggestion, we explored the use of different models to extract image features for clustering e.g. MAE and Dino.

As shown in the above table, there is no big gap in classification performance between using pre-trained model MAE to extract image features and using Dino to extract image features for clustering. As long as with a well-pre-trained model, the image feature for clustering has little impact, all of which can produce better clustering results.

Q4: Corrected notational errors.

Thanks for your kind suggestion, we have modified the paper in detail.

Q5: Normally the NCE contrastive loss only holds one positive label. But for this approach, there could be at least three positive labels. Analysis of Eqn.1.

The NCE contrastive loss only holds one positive label. According to SE in DS-CLIP, one image has $M+1$ multiple generated texts represented as in Eqn.1, which constructs $M+1$ image-text pairs. In Eqn.1, for each image corresponding to $M+1$ multiple texts, we calculate the mean of contrastive loss of $M+1$ image-text pairs during training. In all, Eqn.1 can be interpreted as the mean of multiple NCE contrastive losses.

Q6: Generating images as augmentation is used for DS-CLIP.

The pre-trained dataset for DS-CLIP is YFCC15M, whose text description is too short and unable to adequately describe the image content. The image generated according to the description of the short text is quite different from the original image, which leads to the inconsistency of the sample being self-supervised during the training. The generated images are shown in the following link. generate_img

Training with the generated images in DS-CLIP instead of the augmented image through a transformer, and other modules and parameters are the same with Experiment ID 6 in Table 1 in the paper, the performance decline is more serious(from 39.3% to 30.1%).

Thank you for your insightful advice and valuable questions, we will respond to your concerns point by point.

Q1: The clustering, re-sampling, and text re-generation are all very common strategies in recent work,e.g., BLIP [1], and BLIP-2[2]. Hence the technical contribution is weak.

BLIP[1] proposes an efficient utilization method of web noisy data. It is first trained with noisy data, and then BLIP is used to generate a series of subtitles through a pre-trained Captioner, moreover, the generated subtitles are filtered through a pre-trained Filter, deleting the noisy subtitles from the original web and synthetic texts to get clean data. Finally, BLIP is trained again with clean data. The re-sampling in BLIP is filtered and text re-generation is iteratively updated as model training.

In contrast to the BLIP, our method uses the rich knowledge of LLM/VLLM to enhance the text data and rewrite the data semantically and structurally without repeated model training. We pay more attention to constructing a more diverse and semantic dataset from the web noisy dataset rather than the training strategy. The re-sampling is without filtering data and keeps the diversity of the dataset at each training epoch. Additionally, we conducted experiments comparing the training approaches of initializing the model with the backbone of the VLLM itself and training the multimodal representation model using the data generated by the LLM and VLLM. We find that utilizing the augmented data from a well-trained LLM and VLLM yields superior performance, further confirming the importance of LLM and VLLM in data construction. We hope our augmented dataset will help further research and pay more attention to the data construction by LLM and VLLM.

BLIP-2[2] proposes an efficient pre-trained algorithm through training Q-Former to align a visual model and a Large Language Model (LLM) of freezing parameters. The Q-Former connects an image encoder of a visual model and an LLM encoder. In BLIP-2, the data is not extended or augmented from source, and the training parameters are reduced from the model perspective for efficient training.

The essential difference with our method is that we start from the misaligned between image and text, and use the rich knowledge of LLM/VLLM to enrich text representation. We focus on the problem of data noise from the source rather than the training strategy. We utilize Data De-Duplication ($\text{D}^3$) based on augmented diverse data to achieve superior performance with fewer computing resources. $\text{D}^3$ is different from data cleaning, we do not filter data during training but sample a certain percentage of data points from each pre-clustered center. The text re-generated and sampling strategy are quite different from BLIP[1] and BLIP-2[2]. Extensive experiments show that the constructed dataset is helpful in improving the performance of the current modal e.g. DeCLIP, LaCLIP. Please refer to the question 6 of review YX2M.

Reference:

[1] Li J, Li D, Xiong C, et al. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation[C]//International Conference on Machine Learning. PMLR, 2022: 12888-12900.

[2] Li J, Li D, Savarese S, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models[J]. arXiv preprint arXiv:2301.12597, 2023.

Q2: The previous contrastive loss function can deal with the multi-positive image-text pairs during training. More analysis to verify the effectiveness of the proposed loss function.

The previous contrastive loss function deals with the multi-positive image-text pairs during training as follows.

In BLIP[1], each batch ensures only one positive sample during training.

In ALBEF[3], if there are K-positive samples in batch, then the gt at the corresponding position of contrastive loss is $1/K$.

In our DS-CLIP, for each image corresponding to $M+1$ multiple texts, we calculate the mean of contrastive loss of all $M+1$ image-text pairs.

In our ablation study as shown in Experiment ID 4 in Table 1, only one text is sampled from multiple generating texts with the standard text-image contrastive loss as same as BLIP[1].  Experiment ID 5 in Table 1 shows that multiple text embeddings for DCTM bring a +0.6% improvement in the accuracy of classification. We add an experiment using the contrastive loss in ALBEF[3] instead of DCTM, other parameters are kept the same. The performance with contrastive loss in ALBEF[3] is 38.0% on image classification, which has a comparable result with BLIP[1] as Experiment ID 4 in Table 1. This analysis is presented in Sec4.2.

Reference:

[3] Li J, Selvaraju R, Gotmare A, et al. Align before fuse: Vision and language representation learning with momentum distillation[J]. Advances in neural information processing systems, 2021, 34: 9694-9705.

Thanks to the author for the reply. I carefully read the author's response and the comments of other reviewers. My main concern, similar to reviewer AY1m's, was not well addressed. Current explanations about novelty are still weak. So I changed my rating to 5.

Thank you for your insightful advice and valuable questions, we will respond to your concerns point by point.

Q1: Limitation of the novelty of the proposed methods. Different contributions of our method compared with BLIP and LaCLIP.

BLIP[1] proposes an efficient utilization method of noisy web data. It is first trained with noisy data, and then BLIP is used to generate a series of subtitles through a pre-trained Captioner, next, the generated subtitles are filtered through a pre-trained Filter, deleting the noisy subtitles from the original web and synthetic texts to get clean data. Finally, BLIP is trained again with clean data.

In contrast to the BLIP, our method uses the rich knowledge of LLM/VLLM to enhance the text data and semantically and structurally rewrite the data without repeated model training. We pay more attention to constructing a more diverse and semantic dataset from the web noise dataset rather than the training strategy. Additionally, we conducted experiments comparing the training approaches of initializing the model with the backbone of the VLLM itself and training the multimodal representation model using the data generated by the LLM and VLLM. We find that utilizing the augmented data from a well-trained LLM and VLLM yields superior performance, further confirming the importance of LLM and VLLM in data construction.

LaCLIP[2] rewrites and changes the structure of text from the original text using LLaMA and chatbots but maintains semantic information of the text. Text rewritten in LaCLIP only ensures the semantics associated with the corresponding original text but it does not address the misalignment problem of image and text.

We can supplement the shortcomings of LaCLIP from the aspects of text augmented via VLLM. We combine our generated data with LaCLIP, and the average mAP on 10 image classification datasets is improved by 0.6% with our augmented data. Please refer to question 6 in the review YX2M.

Reference:

[1] Li J, Li D, Xiong C, et al. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation[C]//International Conference on Machine Learning. PMLR, 2022: 12888-12900.

[2] Fan L, Krishnan D, Isola P, et al. Improving CLIP Training with Language Rewrites[J]. arXiv preprint arXiv:2305.20088, 2023.

Q2: Explanation of the effectiveness of using both LLaVA and LLaMA. Verify that it is possible to use LLaVA only to generate diverse captions with proper prompts.

Thanks for your suggestion. The pre-trained dataset for DS-CLIP is YFCC15M in our experiments, whose text description is too short, and the generated texts according to LLaMA have fewer variations. Using LLaMA to generate text has less improvement on YFCC15M.

1. We also pre-train DS-CLIP based on ViTB/16 on CC3M with SE as shown in the following table. The text description of CC3M is longer and more abundant. The improvement is respectively 4.7% and 5.8% based on LLaMA and LLaVA. On the basis of LLaVA, the performance continues to increase by 1.6% combined with LLaMA.

2. We design several proper prompts for LLaVA to generate texts such as ''Can you explain this image in detail?'', ''Tell me in detail what is main object description in this picture.'' and ''What happened in the picture?'' Using these proper prompts to generate three texts for each image with LLaVA on the CC3M dataset, we conduct an experiment named LLaVA+ in the following table. The performance improves by 0.4% from 21.6% to 22.0% compared with text augmented from LLaVA, but it is still lower than text augmentation from LLaVA and LLaMA. The generated text from LLaVA has already described the image content in as much detail as possible, with more prompts to generate texts has fewer improvements.

Tab.3: Ablation study with text augmentation on ImageNet(IN) for zero-shot image classification (%)

Thanks for detailed explanation.

Comparison with BLIP

--> In my review, my main point was that the concept of leveraging synthetically generated captions is the same as BLIP. In their setup, since they did the first work to use generated captions, it is natural to require a training phase for captioner. It is not author's contribution which do not require repetition step since authors simply assume that pre-trained model like LLaVA is available. Moreover, it is really straightforward leveraginng captions from more recent LLaVA which can generate more descriptive captions (which leverages the power of LLM) is much more effective than captions from BLIP. Since the performance increase is mainly comes from SE, my main concern about novelty and contribution is not resolved yet.

Thanks for your insightful advice and valuable questions, we will respond to your concerns point by point.

Q1: Confusing about abbreviations $\text{D}^3$, SE, DS-CLIP, DCTM, and MSTM in abstract and introduction.

I'm sorry that these abbreviations have confused you.

In this abstract and introduction, DS-CLIP is a simple and efficient training algorithm for multimodel representation learning.

Data De-Duplication ($\text{D}^3$) is proposed to reduce training costs without lossing data diversity.

Specifically, we uniformly sample a certain percentage of data points from each pre-clustered center, producing a small subset of the dataset. Therefore, the duplicate data were reduced according to the above re-sampling at each training epoch.

Semantic Enhancement (SE) is proposed to address the issue of image-text misalignment and enrich textual diversity. Different from LaCLIP only with generated texts from LLM, we augment texts from the Large Language Model (LLM) and Visual Large Language Model (VLLM).

With generated multiple texts, we utilize the Diverse Captions Training Mechanism (DCTM) and Modality Self-enhancement Training Mechanism (MSTM) to learn a superior representation model.

DCTM is proposed to use all the generated captions simultaneously, this is shown to improve the performance in experiments. Concretely, it calculates the mean of the contrastive loss for the image to each corresponding text, and standard contrastive loss for the original text to the original image as shown in Eqn (1) and Eqn (2).

MSTM leverages self-supervision within each modality for self-enhancement, the purpose is the same with DCTM. Similar to DCTM, MSTM calculates the mean of the self-supervision loss for the multiple text pairs, and standard self-supervision loss for the image and its augmented image as shown in Eqn (3) and Eqn (4).

We add more explanation and remodify the logic of these representations of motivation. The revised motivation, core idea, and contributions about these abbreviations are shown in General Response.

Q2: Unclear presentation about Image-to-Text Multi-Positive Contrastive Loss and Text Multi-Positive Self-Supervised Loss in Fig.2 and hyper-parameter choice.

Thank you for your suggestions.

1. The calculated processing of Image-to-Text Multi-Positive Contrastive Loss is as follows: one image has $M+1$ multiple texts and each image with one corresponding text calculates a standard contrastive Loss, then the mean of $M+1$ Multiple Contrastive Losses is the final loss as shown in the second one on the left in Fig.2(c).

   Text Multi-Positive Self-Supervised Loss is as follows: $M$ pairwise text pairs in the multiple texts calculate the standard Self-Supervised Loss, and then the mean of Multiple Self-Supervised Loss is the final loss as shown in the first one on the right in Fig.2(c).

   We have already modified Fig.2 and added these explanations in the revised paper.

2. The hyper-parameter of clustered number $K$ is discussed in the ablation study in Appendix.D, we explore three experiments of different clustering numbers (1000, 10000, 100000). We observed a small improvement in classification accuracy as the number of clusters increased (34.2%, 34.6%, 34.7%), which had little impact. We choose the number of clustering (10000) as default.

   The hyper-parameter of $\alpha$ and $\beta$ are equal to 1 in training loss as presented in Training Settings of Appendix.B.

Q3: Improved representation of Fig.2. e.g. the original image and the augmented image are reversed and the spacing between letters is different.

Thank you for your suggestions. The term "enhancement" in our context refers to the augmentation techniques used in self-supervised training of images. The reversed augmented image is transformed by ''FLIP'' as an example. The details of Fig.2. is modified in the revised version. We also upload the modified Fig.2 to the following link. Figure 2

Q4: Defined Several dataset abbreviations in Sec4.3 rather than Sec4.1.

We appreciate your constructive advice. We move these dataset abbreviations from Sec4.1 into Sec4.3 in the revised paper as follows. ''Evaluations are conducted on 10 widely used visual recognition benchmarks including ImageNet (IN), ImageNetV2 (INV2), CIFAR10 (C10), CIFAR100 (C100), Caltech101 (Cal101), Oxford Pets, SUN397 (S397), Food101 (F101), DTD and Stanford Dogs.''