Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models

We thank the reviewer for acknowledging the comprehensiveness of our work on the pre-training stage across a range of multimodal model types. Below are our responses to the specific questions raised:

1. Differences with Prior Works

We thank the reviewer for raising this point and for highlighting relevant prior work. While we acknowledge the similarities in themes with Betker et al. [1] and Chen et al. [2], we believe our study provides several key contributions that distinguish it from these works:

• Comprehensive Investigation of Synthetic Captions:
  Rather than focusing solely on the dichotomy of short versus long synthetic captions, our work conducts a comprehensive study aimed at first verifying the necessity and value of synthetic captions in pre-training multimodal foundation models. Additionally, we examine their interplay with original AltTexts, providing deeper insights into their complementary roles. Building on this foundation, we explore different synthetic captioning strategies and propose the novel AltText Fusion Caption (AFC) format, which balances the strengths of both short and long captions.

• Key Differences with Betker et al. [1]:
  While Betker et al. focused on specific benchmarks relevant to diffusion models, our study expands the scope by incorporating diverse and recent benchmarks, such as DSG and GenEval. These benchmarks provide broader and more reliable evaluations of synthetic captions. Moreover, we evaluate performance across multiple foundational models (e.g., CLIP, MLLMs, and Diffusion Models), showcasing the wider applicability of our findings.

• Key Differences with Chen et al. [2]:

  • Our study scales the exploration of synthetic captions to a 300M dataset, offering insights into their efficacy at a scale representative of real-world pre-training scenarios.
  • Unlike Chen et al., who primarily analyzed the impact of captions on downstream tasks, our study delves into the pre-training stage, investigating how synthetic captions influence model initialization and subsequent performance.

2. Effect of Training with Captions from LLaVA

We thank the reviewer for this excellent point. We also utilized a public model (LLaVA-13B) to caption 300M images in WIT-300M. Table A9 in the Appendix summarizes the comparison between our captions (generated by a 3B model) and those generated by LLaVA-13B. Our findings show that the model pre-trained on our captions outperforms the one pre-trained on LLaVA-13B captions on 7 out of 9 benchmarks. Additionally, experiments on CLIP demonstrate that our captions, despite being generated by a smaller 3B model, achieve results comparable to or better than those generated by LLaVA-13B.

We will revise the manuscript to move this analysis to the main paper for better clarity and emphasis.

3. Motivation for Creating Custom Datasets

We appreciate the reviewer’s question on this topic. While datasets like [3] (300k short captions) and [4] (15k captions) are valuable for fine-tuning, they are insufficient for large-scale pre-training of multimodal foundation models like CLIP, multimodal LLMs, and diffusion models. For example, OpenAI’s CLIP required 400M image-caption pairs for effective training.

A significant contribution of our work is the development of a scalable captioning pipeline capable of generating data at the scale required for pre-training. This scalability allows us to meet the demands of pre-training multimodal models. In the revised manuscript, we will emphasize this distinction and clarify how our dataset complements and advances beyond existing resources.

---

We will revise the manuscript to address these comments. We thank the reviewer again for helping us highlight the contributions of this work.

Response to Reviewer Comments

We sincerely appreciate the reviewer for acknowledging the comprehensiveness of our work. Below are our responses to the specific questions raised:

1. Generalizability of Results Using Synthetic Data from Other Models

Thank you for raising this insightful point. To address generalizability, we conducted additional experiments using the publicly available LLaVA-13B model to generate captions for the WIT-300M dataset (300M images). The results, as shown below, demonstrate that:

• LLaVA-generated captions improve performance on retrieval tasks but reduce accuracy on ImageNet classification.
• Combining AltText with LLaVA captions yields the best overall performance across both retrieval and classification tasks.

These findings align with the trends observed when using captions generated by our 3B captioner, supporting the robustness of our conclusions across different captioning models. The results from LLaVA-generated captions have already been discussed in the main paper (e.g., Figure 2a), and we will further highlight these findings in the revised manuscript.

2. Short Synthetic Captions Fused with AltText

Thank you for this question. Yes, we experimented with leveraging LLMs to summarize AltTexts and synthetic captions into concise short captions. However, we found it challenging to preserve both the richness of AltTexts and the semantic details of synthetic captions in a single short caption. As a result, this approach was not included in the main paper. We will add a discussion of this observation in the revised version.

Question 3: Linear Probing

We follow CLIP (https://arxiv.org/pdf/2103.00020) to perform linear probing: use the features before the linear projection to the embedding space and then train a logistic regression classifier. Linear probing is a way to measure the quality of embedding learning. The linear probing analysis shows the value of DSC captions, suggesting that the relatively weaker results in the zero-shot setting could be attributed to limitations or biases inherent in the benchmark rather than a deficiency in the captions themselves. We will clarify this explanation in our revised manuscript to address any potential confusion.

Question 4: Demographics

Thank you for the suggestion. While we acknowledge that demographic information about human captioners can be relevant in some contexts, our work primarily focuses on the interaction between AltText and synthetic captions and their impact on multimodal foundation models. The demographic background of the human captioners is not expected to influence the conclusions of this study: we also generated captions based on LLaVA-13B and the results consistently lead to similar conclusions, demonstrating that our findings are independent of the specific captioning techniques used.

Question 5: Token Issue and DSC/DSC+

Thank you for raising this question. While the 78-token limit for DSC aligns with CLIP's constraints, our study is not restricted to CLIP but more foundation models. The DSC/DSC+ formats were intentionally designed to explore the optimal captioning strategies for various multimodal foundation models, each with unique capabilities and token-handling limits. Models like multimodal LLMs (e.g., MM1) can process much longer captions. By proposing DSC+ and AFC, we aim to examine how caption format and detail impact performance in systems that are not constrained by shorter token limits. This allows us to investigate the trade-offs between conciseness and descriptiveness across models with differing architectures and token limits.

Question 6: Reverse 2 Stages

Stage-2-HA comprises high-quality human-annotated captions. Consistent with prior works, we fine-tune the captioner progressively, starting with lower-quality captions (Stage-1-1M) and moving to higher-quality captions (Stage-2-HA) to ensure effective training. Reversing these stages may not align with this progressive refinement strategy. On the other hand, the captioner itself is not the primary contribution of our work. Our focus is on studying the interaction between AltText and synthetic captions and identifying optimal synthetic caption formats for various multimodal foundation models.

Question 7: ANA

Thank you for pointing this out. The key difference lies in our approach to quantifying caption richness. Instead of relying on sentence length, which does not necessarily correlate with the level of detail, we introduce ANA as a more precise metric. Assertions, as discrete factual statements, better capture the richness of a caption and provide a clearer basis for quantifying hallucinations. This distinction allows our method to assess captions with consideration of both richness and hallucination.

We will revise the manuscript to address these comments. We thank the reviewer again for helping us revise/polish the manuscript and highlight the contributions of this work.

We are grateful that the reviewer acknowledged the importance of the problem and comprehensive evaluations. Here is our response to the raised questions.

1. Generalizability of Results

First, we respectfully argue that the concern about generalizability may not fully apply in this context given we are studying the compute-intensive pre-training. Our large-scale pre-training experiments are intensive and far from trivial. Our study focuses on a fundamental question: the role of original AltText and synthetic captions in the pre-training stages of multimodal foundation model, and aim to provide data recipe.

Second, in the recent literature, we found that the data recipe is highly transferable among the same-family models. For example, the recipe of Cambrian-1 [7] is fully transferable to MM1.5 [6]. Therefore, we pick one representative model in each family and study the pre-training stage.

Third, we thank the reviewer for sharing these works [1,2,3]. However, it seems these works are not open-sourced for the pre-training. We will add the discussion on more models in the Future Work in our revised manuscript.

2. Human Annotation

Yes, we also directly use LLaVA-13B to generate captions that are independent of human annotation. We run experiments on both CLIP and multimodal LLM, where we observed the same conclusion as our captions. (See the results in Table A9 in appendix and our reply to Reviewer 5ZLH).

3. Findings Are Not Novel

Thank you for sharing this point. While prior works such as RECAP [4] and SPRIGHT [5] have explored mixtures of AltText and synthetic captions, our study introduces several novel contributions.

• We separately investigate the value of AltText and synthetic captions to better understand their individual contributions. This foundational analysis motivates us to generate diverse types of synthetic captions tailored to different use cases.
• In contrast to [4,5], we propose and evaluate additional caption formats, such as detailed captions (DSC+) and AltText Fusion Caption (AFC), to further study the complementary nature of synthetic captions and AltText.
• Both [4] and [5] focus exclusively on text-to-image generation models, whereas our work considers a broader range of multimodal foundation models, including CLIP and multimodal large language models (MLLMs), extending the scope of analysis beyond generation tasks.

Question 1

We appreciate the reviewer’s thoughtful feedback. Our study aims to explore the optimal synthetic captions for various multimodal foundation models, necessitating the generation of diverse caption types, such as SSC, DSC, DSC+, and AFC. To achieve this, we employ an LLM re-write mechanism to systematically prepare these captions. Regarding performance, it is important to note that our captioner is a relatively compact 3B model. A significant contribution of our work lies in the development of a scalable captioning pipeline capable of generating billion-level pre-training volumes of image-caption data. Also, human annotation can help us prepare a cleaner fine-tuning dataset.

We acknowledge that employing larger and more advanced MLLMs could potentially narrow the performance gap further. However, our focus has been on balancing computational efficiency and scalability, ensuring the feasibility of generating pre-training-scale datasets. On the other hand, we want to highlight that a stronger multimodal LLM may not indicate a better captioner, as discussed in Figure 4 and Sec 3.1. We will highlight these considerations and clarify the rationale behind our design choices in the revised manuscript.

Question 2: Another Captioner for MM1

Thank you for the insightful question. To evaluate the impact of using a different captioner, we conducted additional experiments with a publicly available model, LLaVA-13B, generating captions for 300M images in the WIT-300M dataset. The results below demonstrate that LLaVA-generated captions improve performance in few-shot settings, which aligns with the observations reported in our main paper. Similar trends were observed when testing these captions on CLIP, reinforcing the generalizability of our conclusions across different captioning models.

[6] MM1.5 https://arxiv.org/abs/2409.20566 [7] Cambrian-1 https://arxiv.org/abs/2406.16860

We sincerely thank the reviewer for acknowledging the thoroughness and comprehensiveness of our experiments and analysis, which span three diverse multimodal models. As noted, all our experiments are conducted during the pre-training stage, which requires substantial computational resources and is far from trivial. We deeply appreciate the reviewer’s recognition of the effort and significance of this work.

Below are our responses to the two key points raised by the reviewer:

1. Optimal Ratio

Thank you for the constructive feedback. We agree that determining the optimal ratio of synthetic captions to original AltText is a challenging yet essential problem. Our study primarily focuses on investigating the role and format of synthetic captions in the pre-training stage, addressing two core questions:

1. What is the role of synthetic captions and AltText in pre-training? What is their relationship?
2. What is the most effective format for generating synthetic captions?

We verified that incorporating synthetic captions during pre-training consistently improves the performance of multimodal foundation models across diverse benchmarks. This finding highlights their value as an augmentation to original AltText, enriching the diversity and semantic depth of the training data. Through our experiments, we explored various captioning styles (e.g., SSC, DSC, DSC+, AFC) and identified how these formats impact downstream performance, contributing to a better understanding of optimal data design.

Regarding the optimal ratio, our results reveal that:

• The ratio is task-dependent and context-sensitive, as no universal ratio works for all downstream tasks.
• Extreme ratios (e.g., entirely synthetic or entirely original captions) tend to underperform.

For example, as shown in Figure 6, a ratio between 50-90 yields consistently strong performance for CLIP. We will include this high-level guidance in the manuscript to provide actionable insights for practitioners. By presenting these trends, we aim to offer a framework for informed decision-making, while acknowledging that further ratio tuning may be required for specific tasks or domains.

2. Model/Data Release

Thank you for your comment. Our work primarily focuses on analyzing the role and relationships between AltText and synthetic captions, providing data recipes for pre-training multimodal foundation models. While the development or release of captioning models or datasets is not the primary focus of our study, we recognize the value of sharing such resources. We hope to release the model / datasets and are working with our legal team, but it will upon legal approval of our institution. We hope that our data recipe contributes to advancing the research community's understanding and development of multimodal foundation models.