What If We Recaption Billions of Web Images with LLaMA-3?

---

Q5: Section name.

A5: Thanks for your suggestions, we have updated the section name in the main text.

---

Q6: Challenging tasks evaluation.

A6: Thank you for highlighting these benchmarks. We agree that complex evaluation tasks can better assess a model's capabilities. Following your suggestion, we have conducted evaluations on Winoground.

We can observe that ours consistently and substantially outperforms others.

---

Q7: Why use the raw CLIP text encoder in Section 6?

A7: Firstly, we would like to clarify that using the raw CLIP text encoder can provide a consistent and fair reference point that the research community relies on. We also perform additional experiments in switching to our Recap-CLIP text encoder in T2I training: we find that, while quantitatively there is no metric improvement, we as human observers qualitatively can see that the generated images are slightly better especially in generating text.

---

Q8: Why generation prefers p=0?

A8: Thanks for the question. Firstly, we would like to clarify that p=0.1 achieves the best overall performance across metrics. Compared to CLIP training which is discriminative, T2I generation would typically prefer text with more detailed descriptions—this can help in obtaining a more informative text latent which will be helpful in instructing the overall image generation.

---

Q9: What does “re-caption” denote? Using re-captioned prompts for training or inference or both?

A9: Sorry for the confusion. We have clarified our setting in the main text. Please refer to Section 6 evaluation.

---

Reference

[1] Fan, Lijie, et al. "Improving CLIP training with language rewrites." Advances in Neural Information Processing Systems 36 (2024).

Thanks for the appreciation of our work. We address your concerns below:

Q1: Novelty and in-depth analysis.

A1: Thank you for raising this concern. We would like to stress that our primary focus is on creating Recap-DataComp-1B, which, to the best of our knowledge, is the first publicly available image-text dataset with synthetic captions scaled to the billion level using LLaMA-3. We believe this represents a novel and significant contribution to the multimodal research community. While the concept of recaptioning is not new, scaling it to this magnitude with advanced LLMs has not been seen before. More importantly, this large-scale dataset enables the first public, extensive investigations into training CLIP and T2I diffusion models with high-quality synthetic captions. For example, our results comprehensively demonstrate that Recap-DataComp-1B significantly enhances cross-modal tasks, long-context understanding, and text-to-image generation.

Furthermore, as shown in our added rebuttal experiments, our dataset consistently improves the performance of other CLIP-family models, like LaCLIP [1]. It can also serve as an improved pre-training dataset for LLaVA-family models.

Based on this evidence, we believe that Recap-DataComp-1B is a novel and important contribution to the community, with the strong potential to provide significant benefits to future multimodal research.

---

Q2: Is the training with recaptioning harmful to other aspects, e.g., knowledge?

A2: Thank you for raising this concern. We conduct an ablation study on a 30M subset of data with recaptions generated conditioned on the original captions, which is expected to preserve more knowledge in the original caption than our non-conditioned prompt. From the table below, we can see this strategy improves the performance across all metrics, suggesting that designing a better strategy to preserve original captions’ knowledge (e.g., via prompt engineering) could be a strategy to further enhance our framework. We leave its systematic and comprehensive exploration as a future work.

---

Q3: LongCLIP-Large prefers long-caption.

A3: Thank you for raising this concern. We would like to clarify that our primary goal is to measure the alignment between captions and images, both short and long captions. Most publicly available CLIP models, such as OpenAI-CLIP, are optimized for shorter captions, with typical text encoders trained on inputs of up to 77 tokens and often effectively using fewer than 20 tokens.

As a side note, as reported in Section 4.2 of the main paper, we comprehensively report the alignment results not only from LongCLIP-Large but also from OpenAI-CLIP, GPT-4, and human evaluations. We can see that LongCLIP, GPT-4 and human eval consistently prefer to ours than the original caption.

---

Q4: Recaptioning cost.

A4: We benchmarked the inference speed of LLaVA-1.5-LLaMA3-8B on a TPU-V4 256, achieving a throughput of 382 images per second. At this speed, generating captions for the entire Recap-DataComp-1B dataset would require approximately 29 days with these computational resources. We will include the inference time in the next version.

Thanks for the appreciation of our work. We address your concerns below:

Q1: Pure Recaption CLIP Training and Condition prompt.

A1: Thank you for raising this concern. First, we present the results of 100% synthetic caption training below:

This result presents an interesting phenomenon: training exclusively with synthetic captions hurts CLIP performance significantly. More interestingly, as shown in Table 3 of the main paper, mixing a small percentage of original captions (e.g., 10%) can effectively address this performance issue. This result suggests that original captions still play a key role in CLIP training, which may deserve deeper investigation in future research.

Additionally, we clarify that while recent works like LaCLIP [1] and VeCLIP [2] demonstrate synthetic captions can enhance CLIP training, to our knowledge, no prior work has trained models exclusively with synthetic captions. This makes our trial the first public attempt, highlighting the challenges of such an approach.

Regarding the absence of specific named entities, we conduct an ablation study on a 30M subset of data with recaptions generated conditioned on the original captions (examples provided in Appendix E). The results presented in the table show that the improvement is limited. Also, when training solely on conditional synthetic captions, the performance still significantly degrades, highlighting the interesting challenges of training CLIP models exclusively on synthetic captions.

---

Q2: Compare with MetaCLIP.

A2: Thank you for pointing out this related work. We compared MetaCLIP-L/14 with our Recap-L/14, as shown in the table below. Under the same model size, our Recap-L/14 achieves superior performance across all benchmarks. Notably, on retrieval tasks, our model shows an average improvement of 5.4%. We will include this detailed comparison in the next version of our paper.

---

Q3: Total inference time.

A3: We benchmark the inference speed of LLaVA-1.5-LLaMA3-8B on a TPU-V4 256, achieving a throughput of 382 images per second. At this speed, generating captions for the entire Recap-DataComp-1B dataset would require ~29 days with these computational resources. We will include the inference time in the next version.

---

Reference

[1] Fan, Lijie, et al. "Improving CLIP training with language rewrites." Advances in Neural Information Processing Systems 36 (2024).

[2] Lai, Zhengfeng, et al. "VeCLIP: Improving CLIP training via visual-enriched captions." European Conference on Computer Vision. Springer, Cham, 2025.

Thanks for raising many helpful comments. We address your concerns below:

Q1: Novelty.

A1: We would like to stress that our primary focus is on creating Recap-DataComp-1B, which, to the best of our knowledge, is the first publicly available image-text dataset with synthetic captions scaled to the billion level using LLaMA-3. We believe this represents a novel and significant contribution to the multimodal research community. While the concept of recaptioning is not new, scaling it to this magnitude with advanced LLMs is not seen before. More importantly, this large-scale dataset enables the first public, extensive investigations into training CLIP and T2I diffusion models with high-quality synthetic captions. For example, our results comprehensively demonstrate that Recap-DataComp-1B significantly enhances cross-modal tasks, long-context understanding, and text-to-image generation.

Furthermore, as shown in our added rebuttal experiments, our dataset consistently improves the performance of other CLIP-family models, like LaCLIP [1]. It can also serve as an improved pre-training dataset for LLaVA-family models. Based on this evidence, we believe that Recap-DataComp-1B is a novel and important contribution to the community, with the strong potential to provide significant benefits to future multimodal research.

---

Q2 / Q5: Prompt Ablation - How was the prompt chosen?

A2: Thanks for raising this concern. Following your suggestion, we conducted additional ablation on a subset of 30 million image-text pairs. Specifically, we tested four types of prompts to evaluate their impact on the performance of CLIP models trained on the resulting captions. Examples of recaptions are included in Appendix E, and detailed descriptions of these prompts are provided below:

---

Prompts for Image Captioning

1. Original Prompt

Please generate a detailed caption of this image. Please be as descriptive as possible.

2. Concise Prompt

Please generate a short and clear explanation of the image. Please be as concise as possible.

3. Condition Prompt

Please generate a detailed caption of this image. Please be as descriptive as possible based on original caption: [ORIGINAL DATACOMP CAPTION].

4. Diverse Prompts Pool

• Describe the image concisely as short as possible.
• Provide a brief description of the given image.
• Offer a succinct explanation of the picture presented.
• Summarize the visual content of the image.
• Give a short and clear explanation of the subsequent image.
• Share a concise interpretation of the image provided.
• Present a compact description of the photo’s key features.
• Relay a brief, clear account of the picture shown.
• Render a clear and concise summary of the photo.
• Write a terse but informative summary of the picture.
• Create a compact narrative representing the image presented.

Our findings, presented in the table below, show that the choice of prompt has a clear influence on CLIP model performance. Notably, these newly designed prompts outperform the baseline, with the "diverse prompt" approach yielding the best result.

We emphasize that, even with the most vanilla prompt setup, our recaptioned dataset has already shown superior performance in retrieval tasks compared to models trained on in-house datasets such as WebLI-5B.With the new results above, we are now more confident that our work is valuable for enabling future research to build higher-quality datasets by leveraging the latest advancements, such as new prompt engineering techniques or stronger LLMs.

We will add these results and discussions in the future work section.

Thanks for strongly supporting our paper. We address your concerns below:

---

Q1: Word Distribution

A1: Thanks for your question. In Figure 1, "Word distributions" refers to the frequency distributions of words appearing in the captions from both datasets. Specifically, it illustrates how often different words are used in our re-captioned data compared to the original captions. We also include detailed clarification in Section 4.1.

---

Q2 / Q3: Writing and Related Work

A2 / A3: Thank you for the suggestion. We have updated our abstract and included this related work.

---

Q4: More Evaluation

A4: Thank you for the valuable suggestion. We conducted additional evaluations of our CLIP-L/14 model using the Datacomp evaluation tool [1] and compared it to the same model size (L/14) released in [1]. The average scores are presented in the table below. Our model achieves superior performance on 23 out of 38 datasets, particularly excelling in distribution shift and retrieval tasks. These results will be included in our next version.

---

Q5: LLaVA Training Mix

A5: Thank you for the insightful suggestion. Following your advice, we replaced the pre-training dataset of LLaVA 1.5, which originally consists of 558K concept-balanced image-caption pairs from LAION, CC-3M, and SBU, with randomly selected 558K samples from our Recap-DataComp-1B dataset. To ensure a fair comparison, we only updated the pre-training data while retaining the same pre-training recipe. The fine-tuning stage is exactly the same as LLaVA-1.5. As shown in the table below:

We observed that pre-training LLaVA on our synthetic dataset results in a non-trivial performance boost, highlighting the quality and scalability of our dataset. Notably, these results were achieved with random samples from Recap-DataComp-1B without any data curation, suggesting that further improvements are possible with a data filtering process.

---

Q6: Dataset Statistics

A6: Our download date for this version is November 2023, with a total of 940,891,257 valid images successfully downloaded after de-duplication and validity checks. The download success rate for valid images stands at approximately 67.8%. Detailed statistics will be provided in the next version.

---

Q7: Prior Work (Nguyen et al.)

A7: Thanks for correctly understanding the major contribution of this paper! While highly related to prior recaption works, our focus is on creating such a dataset at scale using an advanced LLM and conducting a comprehensive study on how this dataset benefits the training of various image-text models.

Reference

[1] Gadre, Samir Yitzhak, et al. "Datacomp: In search of the next generation of multimodal datasets." Advances in Neural Information Processing Systems 36 (2024).

---

Q2: Pure Recaption for CLIP Training (Reviewer XLPB, TPv5, and mTNK)

A2: We appreciate the reviewers raising concerns about training on purely synthetic captions. First, we present the results of 100% synthetic caption training below:

This result presents an interesting phenomenon: training exclusively with synthetic captions hurts CLIP performance significantly. More interestingly, as shown in Table 3 of the main paper, mixing a small percentage of original captions (e.g., 10%) can effectively address this performance issue. This result suggests that original captions still play a key role in CLIP training, which may deserve deeper investigation in future research.

Additionally, we clarify that while recent works like LaCLIP [1] and VeCLIP [2] demonstrate synthetic captions can enhance CLIP training, to our knowledge, no prior work has trained models exclusively with synthetic captions. This makes our trial the first public attempt, highlighting the challenges of such an approach.

To address the concern about relatively low usage of recaptions in our experiments, we conducted additional evaluations with enhanced training methods like LaCLIP-MT, which leverages multi-positive contrastive loss to train with 100% synthetic recaptions and 100% original captions. The results are presented below:

Note:

• LaCLIP-MT: This is the Multi-Text version of LaCLIP as described in Section 5 of the original paper. In implementing the multi-positive loss, we always include one original caption and one synthetic caption.

As shown, utilizing 100% synthetic recaptions with LaCLIP-MT leads to substantial improvements in retrieval benchmarks, demonstrating the utility of our dataset. While there is a slight decrease in ImageNet-1K zero-shot classification accuracy, this is consistent with observations in prior work [2,3] and highlights the need for further research into effective training techniques with fully synthetic captions.

Additionally, as suggested by Reviewer 47Fv (Q5), we replaced the pre-training dataset of LLaVA 1.5 [4], which originally consists of 558K concept-balanced image-caption pairs from LAION, CC-3M, and SBU, with randomly selected 558K samples from Recap-DataComp-1B. To ensure a fair comparison, we only updated the pre-training data while retaining the same pre-training recipe. Fine-tuning was performed as in LLaVA-1.5. The results are:

We observed a non-trivial performance boost when pre-training on our synthetic dataset, indicating its potential as a valuable resource. These results were achieved with random samples from Recap-DataComp-1B without data curation, suggesting further improvements are possible with a filtering process.

We hope these new results address the reviewers' concerns and increase confidence in the quality and value of our proposed dataset.

Note: All manuscript modifications are highlighted in blue. We are happy to address further questions or concerns during the rebuttal period.

Reference

[1] Fan, Lijie, et al. "Improving CLIP training with language rewrites." Advances in Neural Information Processing Systems 36 (2024).
[2] Lai, Zhengfeng, et al. "VeCLIP: Improving CLIP training via visual-enriched captions." European Conference on Computer Vision. Springer, Cham, 2025.
[3] Zhang, Beichen, et al. "Long-CLIP: Unlocking the long-text capability of CLIP." European Conference on Computer Vision. Springer, Cham, 2025.
[4] Liu, Haotian, et al. "Improved baselines with visual instruction tuning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.