Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning

Thanks for your good questions and constructive suggestions.

Q1: In Tab. 5, LCL is on par with CLIP baseline solely with image-text pairs but is significantly better when using the MMC4 dataset. Whether this performance gain is from the increased number of training samples.

A1: It is reasonable that our LCL is comparable to CLIP on image-text pairs. Our LCL uses the simply constructed LAION-Random dataset (see Section 4.1), degrading to using half of the data for contrastive learning and the other half for generation. Compared to CLIP which uses all data for contrastive learning, there is no significant advantage or disadvantage.

When incorporating MMC4 data, the significant improvement comes from increased data diversity. As mentioned in Section 4.3, all models in Tab. 5 are exposed to the same total number of training images. The number of images per epoch doubles when using LAION-400M + MMC4. Tab. 3 also shows that using MMC4 and LAION-400M alone are on par. This suggests that data diversity is the primary factor.

Therefore, the advantage of our LCL is its ability to use larger-scale and more diverse interleaved image-text data, while not suffering performance losses when using paired data.

Q2: In Tab. 3, using original interleaved datasets such as Obelics does not yield performance gain. MMC4 uses CLIP score to filter image-text pairs and create interleaved data. Any insights on the differences between these two types of interleaved datasets?

A2: This is a very good question, and we suppose that different types of interleaved data may be suitable for different training tasks. Our results suggest that using filtered interleaved data like MMC4 may be better for vision model pre-training. We note that recent work OmniCorpus[a] shows that using original interleaved data like Obelics may be better for fine-tuning MLLM (see their Tab. 4).

[a] OmniCorpus: An Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text. arXiv:2406.08418.

Q3: The scaling behavior is not demonstrated.

A3: We are also very interested in the scaling behavior of our method. We anticipate using datasets such as Laion-2B and OmniCorpus[a] (a large-scale interleaved dataset) to train our vision model to achieve SoTA performance. However, using a total of 34B samples like OpenCLIP, training a ViT-L/14 model would require at least 8000 GPU days, which is more than we can afford now.

Q4: Show the seen samples of each model.

A4: The number of seen images is 13B for all models in Tab. 5 and 6, as mentioned in Section 4.3. For ablation studies, all models use 2B training images, as stated in Section 4.1.

Thank the authors for their response. And most of my concerns have been well-addressed. I'd like to maintain the rating in the current stage.

We thank the reviewer for the reviews and questions. But there is a misunderstanding here.

First, as discussed in Fig. 1, we would like to clarify that our proposed LCL aims to pre-train a vision encoder from scratch using interleaved image-text data, rather than incrementally training a multi-modal model based on a pre-trained vision encoder. Their differences are illustrated in Fig. 1(b) and 1(c) and also acknowledged by Reviewer ktxP.

Accordingly, our experiment is to evaluate the quality of pre-trained vision encoders rather than to compare the performances of different MLLMs. Therefore, when comparing with other pre-training methods, instead of using their released MLLMs checkpoints, we use their released codes or reproduce their methods to pre-train the visual encoders from scratch in a fair setting. We ensure a fair comparison in the same settings, including the model structure, training data, and evaluation settings.

Q1: (a) For BEiT3, the numbers reported are different from the ones reported in the paper. (b) Why the comparison with BEIT3 is only shown in Tab.1 when they report results on VQAv2? The same for CoCA?

A1 (a): The number of BEiT3 comes from training from scratch with BEiT3's pre-training task rather than the reported ones of the trained checkpoint. Tab. 1 compares pre-training methods and pre-training tasks, so the model and data for all methods should be fair. To avoid misunderstanding, we will clarify in the caption of Tab. 1 that the method names in parentheses refer to the pre-training tasks but not their trained checkpoints.

A1 (b): Table 6 evaluates pre-trained vision encoders on multimodal dialogue tasks by integrating them with the same pre-trained LLM for fair comparison. Therefore, the reported results of BEiT3 and CoCa's released MLLM checkpoints are not included. In Table 1, BEiT3 and CoCa are compared as pre-training methods under fair settings.

Q2: In Tab 6, more multi-modal LLMs need to be included.

A2: As mentioned above, our proposed LCL pre-trains vision encoder from scratch, which is orthogonal to the training of MLLMs. Accordingly, our experiment is to evaluate the quality of pre-trained visual features rather than to compare the performances of different MLLMs.

Tab. 6 is to fairly compare different pre-trained vision encoders in multimodal tasks, so the results of other MLLM's trained checkpoints, such as EVA[a], X-Decoder[b], and Emu[c], should not be incorporated. We have used LLaVA-1.5[d] for this evaluation (see Appendix A.2, "Multi-modal dialogue"). There are better MLLMs available than LLaVA-1.5, but training better MLLMs is beyond the scope of this paper, and we do not need to make such comparisons.

In addition, we have compared the text generation + image feature regression task used by Emu in Tab. 1. However, EVA requires pre-trained CLIP features and X-Decoder uses segmentation data, so their training tasks are not comparable to those methods pre-training from scratch only using image-text data.

[a] Eva: Exploring the limits of masked visual representation learning at scale. In CVPR, 2023

[b] Generalized decoding for pixel, image, and language. In CVPR, 2023.

[c] Generative multimodal models are in-context learners. In CVPR, 2024.

[d] Improved baselines with visual instruction tuning. In CVPR, 2024.

Q3: Why is OpenAI CLIP greyed out?

A3: We aim to compare vision pre-training methods rather than pre-trained checkpoints. Both OpenCLIP and we use the LAION-400M dataset for training, allowing for a fair comparison. OpenAI CLIP, which uses private data, is included only as a reference.

We thank the reviewer for the careful reviews and constructive suggestions. We answer the questions as follows.

Q1: (a) The proposed objective seems similar to CoCa, which also employs a contrastive loss and a next token prediction loss. Clarify the differences and why the formulation is novel. (b) In "relation to previous pre-training tasks", why CoCa is not mentioned given the similarities.

A1 (a): CoCa cannot be extended to interleaved data since its learning objective relies on paired data. Applicable to interleaved data is a contribution of our proposed learning objective. There are significant differences between ours and CoCa in how the loss is applied:

1. CoCa's contrastive loss relies on the positive pairs from paired data, which cannot be directly obtained in general interleaved data.
2. CoCa's generation can only be conditioned on a single image, while our generation is conditioned on preceding context containing flexible interleaved image-text contents.

A1 (b): In "relation to previous pre-training tasks," we aim to claim that existing pre-training methods cannot train visual features from scratch using interleaved data. CLIP is the most representative method for paired data, and CoCa is also specialized for paired data. Therefore, we did not specifically discuss CoCa in this context. Given the similarity in the loss form, we will include further discussions about CoCa in the revision and clarify our contribution as mentioned above.

Q2: Equation 3 seems to appear in prior work. Make the attribution to prior work more clear before introducing the equation.

A2: Thanks for your suggestion. Prior work (i.e., M3I mentioned in the paper) showed the relationship between cross-entropy and mutual information in other pre-training tasks, e.g., image self-supervised learning and contrastive learning. While Equation 3 specifically demonstrates this relationship when using an autoregressive model to compress interleaved image-text data. Our novel insight is that, in this scenario, the compression (cross-entropy) objective and the maximum mutual information objective are equivalent. In the revision, we will make the attribution to prior work and clarify our novel insight clearer.

Q3: Make sure that it is trained on sequences with more than one image per sample. Any special tricks to get multi-image to work.

A3: Yes, we support interleaved data containing multiple images. As described in Appendix A.1, each MMC4 sequence may contain 1 to 6 images. We did not use any special implementation tricks. See Fig. R2 in the rebuttal pdf for illustration. The visual features of multiple images are inserted at their positions in the sequence, combined with text embeddings, and then processed through a causal language model to compute the generation loss. For each image, its global feature and the language model’s output feature at its preceding <boi> token form a positive pair in contrastive loss. We will add this information to Section 3.3.

Q4: Why are the numbers for Flamingo in Tab 1 for IN-1k so low?

A4: The number of Flamingo comes from training from scratch with the Flamingo's pre-training task rather than its released checkpoint. Tab. 1 compares pre-training tasks, so the model and data for all methods should be fair. To avoid misunderstanding, we will clarify in the caption of Tab. 1 that the method names in parentheses refer to the pre-training task but not their trained checkpoints.

Q5: (a) Is the COCO CIDEr evaluation protocol zero-shot? The number in Tab.4(a) of 87.5 looks extremely high. (b) Why is Gen. only worse than Con. only for captioning? How is contrastive learner able to do captioning?

A5 (a): The results in Tab. 4 are evaluated under frozen transfer setting, which we will clarify in the table caption. See Appendix A.2 for frozen transfer details. Frozen transfer refers to fine-tuning downstream task models while the parameters of the pre-trained visual encoder are frozen. We use LAION-COCO for captioning fine-tuning, so the COCO CIDEr is reasonable.

A5 (b): The vision encoder pre-trained by "Con. only" also performs captioning by frozen transfer (see Appendix A.2). "Gen. only" is worse than "Con. only" for captioning, because visual features pre-trained by "Gen. only" experienced a collapse. We have discussed this feature collapse in Section 3.2, "Relation to Previous Pre-training Tasks."

Q6: In the frozen transfer setting in Tab. 6, are all models fine-tuned on the same data? If so, what data?

A6: Yes. The frozen transfer in Tab. 6 evaluates the pre-trained vision encoders on multimodal tasks following LLaVa-1.5. All models use the same data and training settings as LLaVa-1.5. See Appendix A.2 "Multi-modal dialogue" for more details.

Thanks for your good questions and constructive suggestions.

Q1: In some cases, the textual context may have little relevance to the image. It is worth investigating whether such data could harm the model's performance.

A1: This is a very good question. It is inevitable that image-text data obtained from the internet may have weak correlations. For fair experiments, we currently follow the settings in previous works when using datasets. Previous works have used filtering methods to improve the quality of data. For example, the LAION-400M dataset used CLIP scores to filter image-text pairs, and OpenFlamingo used CLIP scores to filter out uncorrelated images in the MMC4 dataset. We now follow their settings.

We note that recent work, OmniCorpus[a], analyzed model pre-training with differently filtered data in Section 5.2 (Table 3). The results show that appropriate filtering can improve the correlation of images and texts, thereby enhancing model performance. However, excessive filtering may lead to insufficient data diversity, thus hurting performance. We will analyze the impact of data quality on our model in future work.

[a] OmniCorpus: An Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text. arXiv:2406.08418.

Q2: Qualitative experiments to further demonstrate the effectiveness of the method. Designing reasonable visualization analyses to further elucidate the advantages of the approach.

A2: We use t-SNE to visualize the learned features using images from some classes in ImageNet-1K val set. See Fig. R1 in the rebuttal pdf. For both CLIP and our LCL, visual features of different classes are generally distinguishable, and the distance between classes is related to semantic similarity. Compared to CLIP, visualized points of LCL are slightly tighter for semantically similar classes and slightly further away for classes with low semantic relationships.

Q3: Similar to CLIP, further demonstrating the model's transfer learning performance through domain adaptation tests and few-shot metrics.

A3: Thanks for your suggestion. We adopt zero-shot domain adaptation tests in CLIP. Pre-training ViT-B/16 on LAION-400M, OpenCLIP achieves 67.1 on zero-shot ImageNet-1k, while ours is 60.1. However, domain adaptation via zero-shot retrieval is designed for contrastive learning such as CLIP. It may not be suitable for other types of pre-training methods (e.g., MIM, generative tasks). We have discussed this in Section 4.1.

Moreover, similar to CLIP, we have used probing (fine-tuning a task head with backbone frozen) to evaluate transfer learning capabilities. See Tab.5, IN-1k frozen transfer results. Our LCL is comparable with OpenCLIP, demonstrating the strong transferability of our pre-trained vision model.

We respond to the common concerns of the reviewers here.

First, we claim that our proposed method is the first to use interleaved image-text data to pre-train vision encoder from scratch. Existing vision pre-training methods do not support interleaved data. Simple ways to use interleaved data, e.g., autoregressive text generation, do not perform well for vision pre-training due to the vision feature collapse. We address this by our novel training objective that leverages both contrastive and generation loss.

Q1: Comparison with CoCa's objective.

A1: The major difference between our approach and CoCa's objective is that CoCa does not support interleaved data. For a detailed discussion, please refer to the response to Reviewer A7ig. In the revision, we will further elaborate on the differences between our method and CoCa, both in the figures and the textual descriptions.

Q2: Comparison with state-of-the-art methods.

A2: We would like to emphasize that we propose a vision pre-training method with image-text data. Accordingly, it is appropriate to benchmark it against SoTA vision pre-training methods that use image-text data. We suppose that achieving SoTA performance at the system level of MLLM is not a requisite for a vision pre-training method. Below are two tables revised from the originally tables in our paper, which we will use to further explain our comparison with SoTA performance.

1. Existing vision pre-training methods only support paired data, and we compare with their SoTA performance on paired data. Due to the unavailability of some data, checkpoints, and source code, we have had to either reproduce existing methods or rely on available open-source implementations for a fair comparison (marked by * in Tab R1). The results in Tab. R1 show that our method is comparable to existing methods on paired data.

2. To show that using interleaved image-text data for vision pre-training is not trivial, we also consider existing ways to use interleaved data. Those methods are proposed as MLLM training methods, but we test them for vision pre-training. Tab R1 shows those methods do not work well for vision pre-training.

3. To further confirm our proposed method, we need to compare our pre-trained model with existing checkpoints. Tab. R1 shows that our pre-trained ViT-L/14 is on par or better than the OpenCLIP's checkpoint in a fair comparison. Other checkpoints listed in Tab. R1 could be the reference. They either use a larger vision encoder, more training data, or private data.

4. For multimodal benchmarks, as explained in the response to reviewer KZWZ, our goal is to fairly evaluate pre-trained vision encoders, not to compare different MLLMs at the system level. It is reasonable for us to adopt the open-source LLaVA-1.5. We also list some advanced MLLMs for reference in Tab. R2. Those MLLMs may outperform ours due to their use of stronger vision encoders, larger LLMs, or private data. We anticipate constructing a SoTA MLLM system in future work, but in the current stage, we focus on vision model pre-training due to resource constraints.