SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?

Thank you for recognizing the clarity of our writing and the advantages of using a synthetic dataset for controlled and scalable data collection. We also appreciate your acknowledgment of our ablation studies as insightful and impactful.

Main Claim is Not Demonstrated:

We believe there is a misunderstanding. The main claim of the paper is not to say that synthetic data is superior to real data in all aspects. The paper tries to understand the strengths and limitations of using a fully synthetic pipeline for training CLIP models. We show that on the same scale synthetic data is unable to outperform real data, but upon scaling the gap is reduced and models trained on purely synthetic data, yet with a lower sample efficiency, can outperform CC12M training. This is a novel contribution, since it is unclear how synthetic data representing a large set of concepts would perform in training, and how much they will underperform compared to real data. We also show two hybrid approaches: (1) Fine Tuning a model trained on pure synthetic data on a few real samples, and (2) Training from scratch on a hybrid dataset (synthetic + real samples). These approaches also quantify how much pre-training on synthetic data will differ from joint training in real and synthetic data.

Novelty:

We respectfully but strongly disagree. First, let us highlight that though our concept bank is adopted from MetaCLIP, using 500 thousand concepts for generation allows us to draw significant insights on the impact of distributions in training. Existing works [1,2,3] all assume knowledge of the evaluation benchmarks and heavily bias the generated data to align with the downstream evaluation classes. We also studied various selections of the concept bank such as selecting a random subset of concepts or limiting the concept bank to be the concepts seen in CC3M dataset (Table 4). Our results show that limiting the generation to CC3M concepts (40K concepts) gives you a boost in performance over generation for all 500K concepts of MetaCLIP. On the contrary, generating and training on a random 40K subset (i.e same scale as CC3M concepts but random) leads to lower results compared to 500K concepts of MetaCLIP. Those insights are novel and have not been explored previously in other papers. We also note the following works, published in top-tier conferences, that did not introduce new components but relied on smartly connecting them together or prompting [4,5]. We kindly ask you to reconsider your assessment.

Human Intervention:

While naive data curation pipelines may still lead to usable datasets, these strategies are highly suboptimal and might still violate regulatory policies and might contain prohibited content such as “child abuse” which was found in LAION-5B dataset [6]. Let us also stress that simply recaptioning while ignoring distribution balancing may not fully solve the curation problem, as there are many works relying on advanced data curation pipelines that involve either computational or engineering complexity [7,8,9,10]. Ultimately, we agree to specify this in the paper, but we ask you to reconsider the importance of the property of synthetic data as regards generation of curated data at scale.

Fairness in Comparisons:

This is a common setup in self-supervised learning, and many considerably cited approaches just focus on absolute performance disregarding computational costs [11,12]. Moreover, we observed that CC3M and CC12M datasets reached its highest performance well before completing 40 epochs, with CC12M plateauing at epoch 29. In contrast, SynthCI-30M continued to show improvement up to the 40th epoch. To further investigate this, we conducted an additional experiment using SynthCI-3M, training it for 100 epochs to match the total sample exposure of SynthCI-7.5M trained for 40 epochs. The results showed early performance saturation, with the highest zero-shot accuracy (9.8%) occurring at epoch 24. This experiment suggests that extending training duration on a smaller dataset may not yield any benefits, proving the validity of our comparison.

Real vs Synthetic Data:

Although synthetic data falls short of real data in transfer effectiveness on real-world datasets, we emphasize that uncurated web datasets like LAION and DataComp could face stricter regulatory scrutiny compared to synthetic data, given the growing attention from regulatory bodies. By examining the characteristics of training on synthetic data, we aim to lay the groundwork for developing new foundation models that are entirely free from real data, and thus avoid such regulatory challenges. The performance gap between synthetic and real data is well-documented and widely acknowledged, with extensive literature exploring the distribution shift between the two. While investigating the underlying causes of this gap is an interesting direction, it remains an unresolved issue despite years of research [2] and falls outside the scope of SynthCLIP. Instead, our approach acknowledges the effects of distribution shift and focuses on studying the unique properties of networks trained exclusively on synthetic data.

References:

[1] Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP, NeurIPS 2022

[2] Visda: A synthetic-to-real benchmark for visual domain adaptation, CVPRw 2018

Thank you for your thoughtful and detailed review. We deeply appreciate your recognition of our synthetic data pipeline design and the comprehensive benchmarks and ablations we conducted. Your insights, especially regarding our conceptual distribution and long-tail experiments, highlight some of the key contributions we aimed to achieve in our analysis.

Scalability of the Method:

We acknowledge that the performance at a larger scale is not fully known. While 30M samples might not be at the scale of existing large-scale real datasets, it does allow for understanding the strengths and limitations of synthetic data for training CLIP models. Through that scale of data, we were able to provide interesting insights such as understanding the effect of the concepts distribution (Table 3), the importance of various modalities (Figure 4), and the effect of choice of language models (Table 3). Our paper serves as a stepping stone for defining possible good practices in synthetic training of CLIP models. We do not claim to solve the problem entirely, however we hope this project, and all the assets (code, data and models) that will be open-sourced to help the research community. Even though M3AE explored a relatively small scale for data, their impact on the community is reflected by the citations on that work.

Delta MTL Calculation:

MTL is a metric that evaluates the relative performance across multiple tasks by normalizing improvements or degradations with respect to baseline performance. Specifically: For each task $i$, given its baseline performance $b_i$, observed performance $m_i$, and direction of improvement $g_i$ ($0$ if higher values are better, $1$ if lower values are better), the relative performance improvement or degradation is computed as:

$\Delta_i = (-1)^{g_i} \frac{(m_i - b_i)}{b_i}$

The final MTL score is the mean of all task-level relative performance scores, expressed as a percentage:

We added this to the revised manuscript.

Training with CC3M Concept Distribution:

While this is an interesting experiment, it would be hard to implement due to the following reasons: (1) Given a caption from the CC3M, deciding on which concept of the many concepts that appear in the caption is the one we generate the synthetic caption and image for is non trivial. (2) Assuming we select the main subject of the caption to the be concept, which will lead to loss of certain concepts, we cannot control which concepts the language model will decide to add to the generated caption, hence the control over the distribution is very challenging.

Instead we explored the impact of concept distribution by comparing performance when training SynthCLIP on CC3M-specific concepts ( $C_{CC3M}$) versus random subsets ($C_{\text{rand}}$). $C_{CC3M}$ was created by identifying approximately 40,000 concepts in our bank that overlap with CC3M captions, while $C_{\text{rand}}$ contained 40,000 randomly chosen concepts. Training on $C_{CC3M}$ led to better performance in tasks like text retrieval ($+3.9\%$) and linear probing ($+1.6\%$), likely due to alignment with downstream datasets, reflecting a distribution bias in CC3M toward commonly evaluated tasks. Conversely, $C_{\text{rand}}$ underperformed across benchmarks, showing the importance of concept relevance.

Log-Scale Error Plots:

Thanks for the interesting suggestion, which led to interesting findings. We performed the experiment and included it into the supplementary with a new plot. Please refer to Figure 8. Overall, we observe that coefficients are similar across tasks for real and synthetic data, respectively. We believe that the main cause of such behavior is the distribution shift, that is impacting the learned representation equally for each task. With future developments of diffusion models allowing for compensating such distribution gaps, it is realistic to assume that the tasks analyzed would be improved.

Long-tail Concepts:

Thanks again for the interesting experiment proposed. We followed your suggestion and evaluated the effective robustness properties of SynthCLIP on different datasets similarly to [1], as also suggested by Reviewer c8nx. We present results in Table 9 in the supplementary material. Overall, we do not observe robustness properties due to synthetic data usage. However, mixing real with synthetic data in a similar fashion to those explored in Table 7 allows us to get best performance. We hypothesize that this is due to the simultaneous compensation of the distribution shift (due to the inclusion of real data) and the improved quality of the representations (due to the quality of synthetic data and coverage of concepts).

Thank you for acknowledging the significance of our pipeline and dataset, as well as the rigor and depth of our experiments. We greatly appreciate your thoughtful review and encouraging feedback.

Depth of Zero-Shot Image Classification Results and ImageNet Variants:

We appreciate the feedback and agree that a more comprehensive evaluation would strengthen our findings. Before presenting additional experiments, let us highlight that our evaluation goes way beyond zero-shot image classification, including also linear probing, few-shot learning, text retrieval, and image retrieval tasks across multiple datasets (refer to Table 1a and 1b). We will highlight the fact that higher zero-shot accuracy correlates with improved performance across tasks.

Following your suggestion, we have conducted further zero-shot evaluations on ImageNet variants such as ImageNetV2, ImageNet-A, ImageNet-R, ImageNet-O, ImageNet-Sketch, and ObjectNet. We will include these results in the revised manuscript.

As shown in the previous table, the performance gap is similar to the one in ImageNet, suggesting that the distribution gap is the most influential source of performance degradation. To further provide intuitions on settings involving both real and synthetic sata, we evaluated in two additional setups: 1) the finetuning setup following the settings of Table 2, and the mixed setup of Table 7 in the appendix. Results are below:

As visible, performance still follow the ones reported in the main paper: mixing data performs best, but finetuning synthetic-pretrained representation extractors already allows for a major boost in performance.

Comparison with LAION-400M:

Thanks for the interesting suggestion. We performed the experiment and trained on a random 3M images subset sampled from LAION-400M. Here, we report the results.

In particular, let us highlight the significant drop in IR and TR due to the usage of LAION. We attribute this to the non-descriptive captions, lacking the data curation of CC3M. This is further proof of the benefits of synthetic data in this case, considering that we allow for improved performance in IR and TR without human data curation as in CC3M.

Performance on ImageNet Variants and MLLM Understanding Tasks:

While MLLM requires more sophisticated textual encoders, and as such training such a model goes beyond the scale of data investigated in our paper, we believe there is a reasonable expectation that the superior alignment between text and images of synthetic data would improve reasoning performance. Throughout the paper we reported results on Image Retrieval (IR) and Text Retrieval (TR) which require knowledge of both text and image, as MLLM reasoning tasks. Our experiments showed that training on synthetic data at scale exhibits the highest IR and TR accuracies (for example Table 1-b SynthCLIP with 30M samples achieve 61.7% on IR and 77.1% on TR compared to 58.9% and 71.7% achieved by CC12M training). Additionally, Table 8 shows the results of fine tuning OpenAI CLIP on our generated synthetic data and we find that the model performance increases on both IR and TR highlighting the effectiveness of synthetic data on vision-language understanding of the model.

Thank you for recognizing the value of our dataset and its potential to enable safer and effective model training, as well as highlighting the comparable downstream performance of models trained on synthetic captions. Your thoughtful feedback is greatly appreciated.

Motivation for Using Purely Synthetic Image-Caption Datasets:

We appreciate this observation and acknowledge the need to clarify our motivations. Our work aims to explore the potential and limitations of synthetic data for pre-training CLIP-like models, rather than advocating exclusively for synthetic data. We think that training on synthetic data will be increasingly important in the next future. Indeed, while collecting data on the web is arguably easy, curating such data is a cumbersome and cost-inefficient operation. Synthetic data, conversely, allow for an automatic data curation, and for having control over the content generated. This not only allows us to benefit from strong composition capabilities, allowing it to hallucinate objects difficult to collect in real life (e.g., an elephant on the moon), but also enable further control over copyrighted and safe content. We believe this will also increase in the future due to the efforts in differentially private [1] and safe [2] diffusion models.

While our experiments demonstrate that hybrid approaches (e.g., fine-tuning on real images after pretraining on synthetic data) outperform purely synthetic models, our primary goal is to understand the standalone capabilities and limitations of synthetic data due to the aforementioned advantages. Let us highlight that in the main paper we included a hybrid approach on small curated real data (Table 2), which is a realistic scenario in presence of a large-scale synthetic dataset, automatically curated.

We updated the abstract of the paper to clarify this.

Faithfulness to Content and NSFW Content in Generated Images:

Although we do not provide guarantees on the generated content, these are open problems in text-to-image generators, with significant efforts in the state-of-the-art to achieve faithful [3,4,5] and safe [2,6] outputs. Since our formulation is general, with further research on the topic, these issues will be solved by the diffusion model used for generation. However, we did our best effort to quantify the effects of misalignment between captions and images, and NSFW outputs. Both impact marginally our results. In particular:

• Faithfulness to Content: We conducted experiments where we recaptioned generated images (Table 4a). This process improved performance across most tasks, demonstrating that enhancing caption post-generation can quality can mitigate content unfaithfulness. However, we managed to scale the training even with original captions.
• NSFW Content: Our dataset analysis revealed that approximately 3.15% of the MetaCLIP 500k concepts are NSFW concepts. To eliminate NSFW generations, we implemented a filtering mechanism to exclude NSFW concepts from our concept bank, thereby reducing the incidence of NSFW content in generated images. This is presented in Section 6. Let us also highlight the uncurated LAION-5B approximately 3% NSFW concepts [7], also containing illegal child abuse images regardless of security checks [8]. This emphasizes the challenge of collecting safe real data at scale, and advocates for the advantages of our synthetic generation. To further support our argument, we applied an NSFW detector [9] to the SynthCI-30M images after filtering, revealing only 0.005% NSFW content. This shows that the impact of safety degeneration is marginal.

Augmenting Real Datasets with Synthetic Images from Tail Classes:

We appreciate the reviewer’s feedback and acknowledge the limitations of generative models like Stable Diffusion in representing tail classes. As detailed in the discussion section, our experiments demonstrate that augmenting datasets with synthetic images improves downstream task performance, even for tail classes. Specifically, we observed significant improvements in zero-shot classification accuracy for 10 tail classes: 44.18% accuracy for CLIP versus 60.04% accuracy for SynthCLIP, with 150 samples per class. We attribute this behavior to the synthesis of parts or patterns typical of those classes, that may be rendered realistically even though the capabilities of Stable Diffusion on such elements are limited. These results illustrate that synthetic data can enhance performance by increasing diversity and coverage, especially in challenging long-tail distributions. While we recognize existing limitations in generative model performance for tail classes, these findings suggest synthetic augmentation is a promising approach.