What If We Recaption Billions of Web Images with LLaMA-3?

Q1: No algorithmic contributions.

A1: While we don’t introduce new recaptioning algorithms, we highlight two core contributions: Recap-DataComp-1B, the first fully open, billion-scale image-text dataset with synthetic captions generated using LLaMA-3. Prior works [1,2] are proprietary and closed-source. Scaling recaptioning with advanced LLMs at this scale is unprecedented and enables reproducible, large-scale research in multimodal learning.

This dataset supports the first public large-scale evaluation of CLIP and T2I diffusion models trained on high-quality synthetic captions. Our experiments show notable gains in cross-modal tasks, long-context understanding, and image generation, making Recap-DataComp-1B a valuable resource for the community.

[1] Improving Image Generation with Better Captions. 2023 [2] A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image. 2023

Q2: Misalignment between their dataset’s advantage and results.

A2: Our main goal is to improve noisy web captions by aligning them better with images. Hence, we focus on tasks like cross-modal retrieval that directly measure image-caption alignment. We also include human and GPT-based evaluations (Sec. 4.2 and Appendix A) to assess caption quality. In classification, training only on synthetic captions degrades CLIP performance. However, mixing just 10% of original captions substantially recovers accuracy, confirming that original captions still play a key role in avoiding data collapse. In contrast, T2I diffusion benefits from detailed, synthetic captions. Setting the mixing ratio to p=0.1 (i.e., 90% recaptions) leads to better generation results since richer captions produce better text latents.

Q3: More complex benchmarks (such as image captioning and VQA). Table 6 is less comprehensive.

A3: Thank you for the suggestion. Our goal is to provide a high-quality, large-scale dataset for both understanding and generation tasks. We conducted large-scale CLIP and T2I experiments to demonstrate its effectiveness and potential. Table 6 is limited by the high computational cost—for instance, training CLIP ViT-B/16 for 2 epochs only takes approximately 0.5 days, while DiT-B/4 requires around 7 days for a single epoch on TPU-v3-256. These constraints restricted the number of benchmarks we could include. Furthermore, the current results already demonstrate our dataset’s effectiveness, and we will leave further evaluations to future work.

Following your advice, we adopted the LLaVA framework to evaluate more challenging tasks such as VQA and image captioning. We replaced its pretraining data (558K LAION/CC3M/SBU) with 558K Recap-DataComp-1B samples. Results show consistent gains, including +0.6% across four VQA tasks and +34 points on MME shown in the following table:

Lastly, we further evaluated the CLIP visual encoder trained with our mix-training strategy on our dataset. Our LLaVA with ViT trained only on synthetic captions (p=0) shows only a slight drop compared to training on the original data (p=1) — an average of 1.2% across four VQA benchmarks (TextVQA, GQA, MM-Vet, SEED). Notably, at p=0.8, LLaVA achieves the best overall performance, matching the p=1 model and surpassing it on MME by 35 absolute points.

These results demonstrate that by evaluating more diverse and challenging tasks, our proposed dataset not only significantly enhances cross-modal retrieval performance but also reveals promising

Q4: Is this data DataComp still publicly available? Safety fixes of LAION.

A4: Yes, DataComp-1B remains public and is not derived from LAION-5B but from a different Common Crawl version. The original dataset is accessible at Huggingface Dataset. More importantly, it applied strict NSFW filtering, and we discussed safety considerations in the Appendix. We believe releasing our recaptioned dataset poses no major safety risks.

Other: on page 5, line 227, we cite the GPT-4V rating. We will clarify this in the revision.

Q1: Novel ideas, insights, and conclusions.

A1: Thank you for raising concerns about novelty. Our primary focus is on developing Recap-DataComp-1B, which we believe to be the first publicly available image-text dataset at the billion-scale generated by LLaMA-3. Unlike previous works [1,2] that remain proprietary—both in terms of captioning models and datasets—our approach is fully open and demonstrably scalable to the billion level. We believe this openness and scale set a new milestone in multimodal research.

While the idea of “recaptioning” is not entirely new, we have yet to see it executed at this scale using advanced large language models. By creating recaptions on such a massive level, our dataset enables the first public, large-scale experiments into training models like CLIP and T2I diffusion with high-quality synthetic captions. In our view, Recap-DataComp-1B offers a novel and significant contribution to the field, with considerable potential to advance future multimodal research.

[1] Improving Image Generation with Better Captions. 2023 [2] A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image. 2023

Q2: Marginal improvement on some tasks.

A2: Thank you for the question. As shown in Table 1, the LLaVA-3 8B model offers slightly better performance than the 13B model while being significantly faster. We selected it to balance captioning quality and efficiency. In Table 4, Recap-CLIP clearly outperforms the previous DataComp-1B baseline trained on the same samples, highlighting the improved quality of our dataset. While our results slightly exceed those of SigLIP, it’s important to note that SigLIP requires four times more training samples on a private dataset, which is five times larger—making our gains particularly noteworthy.

Q3: Weaker performance on ImageNet-1K. Are more tasks harmful?

A3: Thanks for your constructive suggestion. We observe that training exclusively with synthetic captions degrades CLIP performance. However, introducing a small portion of original captions (e.g., 10%) effectively recovers performance, indicating that original captions remain crucial in preventing data collapse, as noted in prior work [1]. Additionally, we clarify that while recent works like LaCLIP [2] and VeCLIP [3] demonstrate synthetic captions can enhance CLIP training, to our knowledge, no prior work has trained models exclusively on synthetic captions. Our study represents the first public attempt, shedding light on the challenges of this approach.

Second, we further evaluate the visual representations learned from these models through linear probing and full fine-tuning on ImageNet-1K. While the CLIP ViT trained only on synthetic captions underperforms in zero-shot classification, We observe that lightweight tuning (e.g., linear probing or fine-tuning) greatly narrows the performance gap seen in zero-shot settings. Moreover, models trained on mixed captions perform on par with those trained on original data, striking a favorable balance between classification and retrieval performance.

Following your advice, we also evaluated additional tasks to analyze the impact of recaption following the original DataComp paper across 22 datasets (VTAB contains 13, IN-1K-Shift contains 6, and Retrieval contains 3), shown in the table below. We found that purely synthetic captions negatively impacted zero-shot performance. However, our mix-training strategy still achieves superior performance in VTAB and retrieval tasks.

These results indicate that only synthetic captions may negatively impact zero-shot classification performance. Our mixed-training strategy effectively balances these competing factors. We acknowledge that further investigation into potential drawbacks is valuable. Given the demonstrated scalability and high-quality alignment of our proposed dataset, we believe it provides a strong baseline for future research.

[1] Seddik M E A, et al. How bad is training on synthetic data? a statistical analysis of language model collapse[J]. arXiv, 2024. [2] Fan, Lijie, et al. "Improving CLIP training with language rewrites." NeurIPS. 2024.
[3] Lai, Zhengfeng, et al. "VeCLIP: Improving CLIP training via visual-enriched captions." ECCV. 2024.

Q1: Analysis of this limitation, insights, and technology to migrate classification performance issues.

A1: Following your valuable suggestion, we analyze the classification drop with synthetic captions and identify three likely causes: (1) CLIP's difficulty learning from long, rich captions [1], (2) captioners’ poor performance on classification tasks [2], and (3) limited evaluation of zero-shot only. We additionally evaluate performance on the standard ImageNet-1K classification task using both linear probing and full fine-tuning. We observe that introducing lightweight tuning significantly reduces the initial performance gap seen in zero-shot settings, indicating that models trained solely on synthetic captions still learn strong visual representations.

Regarding potential strategies, mixed training already achieves a preferable balance of classification and retrieval, and recent work [3] suggests that advanced CLIP training methods can further exploit synthetic captions.

Q2: Caption model and quantitative results.

A2: Thanks for bringing up the concern about ablation on caption models. We evaluate four caption models on a 30M subset and the results are shown in the following Table. We find that LLaVA-1.5-LLaMA3-8B consistently yields the best downstream CLIP performance. Larger models like LLaVA-NeXT-Gemma2-27B are significantly slower—requiring 201 days to recaption the full 1 billion samples on the same infrastructure—and also underperform on retrieval tasks. Thus, we chose LLaVA-1.5-LLaMA3-8B for re-captioning the full dataset, balancing quality and efficiency.

Q3: Prompt engineering and quantitative results.

A3: We ablate prompt engineering on a 30M subset and found prompt choice influences downstream tasks shown in the following table:

Our findings show that even simple prompts, such as the Original setup, already significantly enhance retrieval performance at this scale. As the first to release a billion-caption dataset, these promising results motivate us to further expand and refine our evaluations in future work.

Q4: Limited Human Evaluation.

A4: We present human evaluation results in Appendix A and Figures 4 and 5, which align closely with GPT-4V assessments. We will further emphasize this consistency in future revisions.

Q5: Potential biases and failure cases in the generated captions.

A5: We acknowledge the presence of biases and failure cases in generated captions inherited from the training data of LLaMA-3 and LLaVA. Addressing and mitigating these biases will be a key focus of our future work. Regarding presenting potential failure cases, one example of a current shortcoming is the pipeline’s inability to correctly identify certain objects—for instance, the “Western Kingbird” shown in Figure 1. We will include a more detailed analysis of failure cases in the next version.

Q6: Smaller, curated subset.

A6: We will release the ablation subset to support further research.

[1] Zhang, Beichen, et al. "Long-CLIP: Unlocking the long-text capability of CLIP." ECCV, 2024.
[2] Zhang, Yuhui, et al. "Why are visually-grounded language models bad at image classification?” arXiv, 2024. [3] Liu, Yanqing, et al. "CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captions." arXiv, 2024.

Thank you for recognizing that our dataset significantly enhances performance in both multimodal retrieval and text-to-image generation tasks. We appreciate your insights into how advanced MLLMs can reduce reliance on costly human annotations and enable greater scalability. Our Recap data relies solely on LLaVA-based annotation, which inherently incorporates commonsense knowledge from a human perspective — since LLaVA is an aligned model trained on human preference data. In other words, our Recap approach not only reduces dependence on manual annotation but also opens up promising directions for future research. Our experiments further show that mixed-caption training — integrating both CLIP and T2I methods — offers a robust way to incorporate external commonsense knowledge, leading to consistent performance improvement.

In essence, our dataset serves as a valuable baseline and starting point for exploring these innovative strategies. We plan to expand on the broader impact of this work in the upcoming version.

Q1: Why not use earlier LLaVa-LLaMa3 models?

A1: Thank you for bringing up this work. First, we trained our own version to maintain full control over the training process while adhering to open-research practices and a broader licensing framework. Second, our initial investigation revealed that the original LLaVA-1.5 data consistently started its outputs with "This image is a …". To address this issue, we integrated a high-quality dataset (HQ-Edit) into our training process, which alleviated the problem. We will include this essential reference in our next version.

Q2: Technical innovation.

A2: We would like to stress that our primary focus is on creating Recap-DataComp-1B, which, to the best of our knowledge, is the first publicly available image-text dataset with synthetic captions scaled to the billion level using LLaMA-3. We believe this represents a novel and significant contribution to the multimodal research community. While the concept of recaptioning is not new, scaling it to this magnitude with advanced LLMs has not been seen before. More importantly, this large-scale dataset enables the first public, extensive, and fair investigations into training CLIP and T2I diffusion models with high-quality synthetic captions. For example, our results comprehensively demonstrate that Recap-DataComp-1B significantly enhances cross-modal tasks, long-context understanding, and text-to-image generation. Based on this evidence, we believe that Recap-DataComp-1B is a novel and important contribution to the community, with the strong potential to provide significant benefits to future multimodal research.

Q3: Complexity of creating and training models of our dataset.

A3: We benchmark the inference speed of LLaVA-1.5-LLaMA3-8B on the TPU-V4 256 hardware, achieving a throughput of 382 images per second. At this rate, generating captions for the entire Recap-DataComp-1B dataset (~1 billion images) would take approximately 29 days of continuous computation.

Regarding CLIP training, training a ViT-L/16 model for two epochs (~2.56 billion total samples) on Recap-DataComp-1B requires ~ 1 day on TPU-V3 256 infrastructure. For DiT training, training a base-sized DiT model with a batch size of 2048 for 650K steps takes approximately 7 days using TPU-V3 256 hardware.

We will provide comprehensive details about computational complexity and runtime metrics in the next revision.