Demystifying CLIP Data

Please note that our data is not a reproduction of OpenAI-CLIP (we cannot reproduce because the CLIP paper only has very limited information about its data sources), instead MetaCLIP makes language-image curation transparent so it can be applied to different pools to create high quality datasets which lead to better performance than CLIP, under the exact same training and model settings.

Regarding the mixed performance gains we think it’s due to the following::

• MetaCLIP vs OpenAI CLIP: the mixed performance gains are mainly due to pool sources, but since OpenAI-CLIP’s data source is unknown, it is hard to draw a conclusion;
• MetaCLIP vs LAION/DataComp: since the source is the same, we believe the better performance of MetaCLIP has better coverage of downstream tasks with metadata balancing that favors more tasks by sampling data for long-tailed distribution, whereas LAION/DataComp has no balancing and favors head distribution, or what the OpenAI-CLIP model is biased to (which again is unknown). We hope that with open-sourcing of our full pipeline the community can develop further datasets to study performance across a variety of tasks and further our understanding in language-image pre-training.

how does the curation step proposed in this paper compare with the one used for LAION-400M

MetaCLIP uses raw metadata substring matching and balancing, whereas LAION-400M uses an OpenAI-CLIP filter and text length filters (page 3 of https://arxiv.org/pdf/2111.02114.pdf), as well as hidden filters not mentioned in the paper but in the pipeline code (e.g., filters on relative URLs in parser that can be observed in the code of LAION release).

LAION’s curation is based on an OpenAI-CLIP model to filter the data, therefore it distills the OpenAI-CLIP data implicitly. For example, a model trained on CLIP’s data will directly have high accuracy/similarity on the CLIP training images; therefore, this can be used to replicate CLIP’s data, but it can have low accuracy/similarity on image/text pairs that are not in CLIP’s training set. Instead, our goal is curating data from scratch. We wish to highlight the beauty of our raw curation, without (model-based) filters, to maximally preserve signal and control biases.

can you please explain why your curation is not overfit towards ImageNet?

We didn’t do anything particular for ImageNet besides following OpenAI-CLIP, which uses all WordNet entries that includes ImageNet labels. The metadata contains 500k entries and ImageNet is only 1k of them; therefore, the metadata covers a large, general pool of concepts. We also de-duplicate our data against ImageNet.

We want to note that this approach is a much weaker bias to ImageNet than the “Image-based filtering” used in DataComp, which, as mentioned above, searches the nearest neighbors for every ImageNet training example, and keeps examples belonging to these groups. For example, for their ViT-L experiments, the ImageNet “Image-based filtering” increases ImageNet accuracy from 76.4% to 79.2% (see Tab. 3 of the DataComp paper).

Finally, note that we report strong results on 26 benchmarks in Table 4, and on 38 benchmarks in Table 8; suggesting that our approach does not overfit to ImageNet but is generally working on multiple datasets. We however, do agree that the data distribution matters for downstream performance and performed an ablation in Table 11, where we show the number of classes that have matches in the metadata. For example, for ImageNet 703 of the 1000 classes are present in the metadata, and more general we can see that most datasets have text string matches in the very general and large (500k) set of metadata.

The contribution of the paper is limited. The main contribution is the entry balancing used by CLIP. This balancing operation actually plays a similar role to the deduplication used in [1] and [2], where its effectiveness has been proven.

The contribution of this paper is the full MetaCLIP curation pipeline (that creates a structured dataset based on large-scale metadata), which is novel to the community and allows to have full transparency and control for image-language pretraining. This is developed on top of the limited information present in the CLIP paper (note that the original CLIP paper does not describe the data curation pipeline and only contains three paragraphs about it, in the 48-page (Radford et al, 2021) paper). Besides that, our curation is scalable to the whole Internet and outperforms CLIP’s data by a significant margin.

From the balancing perspective, [1, 2] use trained CLIP embeddings as features for K-means clustering and perform duplicate removal via excluding examples too close to cluster centers. This post-hoc filtering may carry CLIP’s unknown biases (e.g., see discussion in LAION-5B Appendix G.2) and MetaCLIP has balancing via matching unstructured, noisy text. The main difference between MetaCLIP and [1, 2] is that these works are filtering methods that filter data using heuristics and the CLIP model. Therefore, they rely implicitly on the unknown CLIP data. Our method instead directly creates the data in a transparent Algorithm, without reliance on filters or external models.

it is worth trying some model-based rules, e.g., image-text matching based rules. Although the paper mainly aims to reproduce the CLIP paper's results, it would lead to more contributions to go beyond CLIP reproduction and curate higher-quality datasets.

These are great ideas to further increase data quality; however, our goal is to make CLIP curation transparent: using OpenAI-CLIP adds a black-box filter back and thus conflicts with our initial goal. For example, a model trained on CLIP’s data will directly have high accuracy/similarity on the CLIP training images; therefore, this can be used to replicate CLIP’s data, but it can have low accuracy/similarity on image/text pairs that are not in OpenAI-CLIP’s training set. Instead, our goal is curating data from scratch. We wish to highlight the beauty of our raw curation, without (model-based) filters, to maximally preserve signal and control biases.

How many runs are conducted to get the quantitative experimental results in the paper (e.g., Table 4 and 5)? Are the standard deviations sufficiently smaller than the differences between different settings?

Yes, significantly smaller. The standard deviation of multi seed training is +-0.1% acc for ImageNet on ViT-B/32. We added this information to the paper.

Any analysis on where the improvement over CLIP dataset is from in Table 4? Different raw data sources?

Unfortunately, we have no access to OpenAI-CLIP's training data for direct comparison. Yes, we also think it is because of different raw data sources: Our accuracy over the different downstream datasets appears closer to LAION, since it is based on the same CommonCrawl source. We have no information about OpenAI-CLIP’s source, but would be happy to perform more analysis for the final paper.

We thank the reviewer for their insightful review and suggestions for improvement. Please let us know if anything else is unclear and we will try to answer.

The paper is a commendable effort to reproduce the data curation pipeline that has already been described in the original CLIP paper (Radford et al, 2021), and report the findings for reproducing an existing data curation technique.

We would like to note that the original CLIP paper does not describe the data curation pipeline. We are following (and quoting in the submission) all information about CLIP’s data curation, which is extremely limited (three paragraphs in the 48-page (Radford et al, 2021) paper). Our full MetaCLIP curation pipeline presented in the paper is developed on top of this limited information and novel to the community. This has not existed before, instead the community has been using CLIP models to reproduce its data.

Besides this new and scalable curation approach (e.g. Alg. 1 runs at Internet scale of 300+B pairs and performs online curation in a data loader in Table 6) we believe that the scope of novelty is broad and is beyond technical algorithms. It can also include extending existing paradigms to new problems, designing new experiments/ablations, drawing new observations/insights, and verifying expected hypotheses in new problems or tasks. From the reviewer comments, we have seen that many of these values of our work are recognized. We thank the reviewer for seeing value in our work from that perspective.

Page 3"this post-hoc data pruning approach has limited utility, as the computational resources saved have already been expended during the initial training of the model." The reviewer has some concern regarding this comment. While posthoc data pruning still requires training an initial model, the pruned dataset can also allow efficient model improvement, which is usually an iterative process involving experimentation with different model hyperparameters.

We thank the reviewer for pointing this out and rephrased accordingly. Our main point here was that costs associated with training a model for filtering (e.g. a CLIP model) should also be taken into account. We agree on the value of data pruning, especially with open models and known training data.

Is there a specific reason to omit some other data curation baselines proposed in the DataComp paper?

Please note that DataComp is a concurrent work, but we used their data pool and show that MetaCLIP can generalize to it in our appendix. Our experiments with DataComp’s pool are shown in Tab. 4 and Fig. 4 of the Appendix.

We further compare the benchmark used in DataComp in Table 8. These are all apple-to-apple comparisons, using the same hyperparameters and models and therefore directly measure data quality.

A direct comparison to the DataComp filtering is not possible for the following reasons: (1) DataComp uses different hyperparameters than OpenAI-CLIP. (2) DataComp curation is based on two OpenAI-CLIP filters (B and L sizes), therefore it distills the OpenAI-CLIP data implicitly. (3) DataComp performs “Image-based filtering” that requires clustering into 100k groups and finding the nearest neighbor group for every ImageNet training example, and keeping examples belonging to these groups. Having access to CLIP models and ImageNet training data is a non-trivial advantage and it has significant gains as ablated in Tab. 3 of the [DataComp paper](https://arxiv.org/pdf/2304.14108.pdf). For example, for their ViT-L experiments, the “Image-based filtering” increases ImageNet accuracy from 76.4% to 79.2%. Nevertheless, please note that our MetaCLIP system, which does not need external models or data, provides similar performance on the DataComp benchmark, without any bells and whistles.

What is the meta data distribution for OpenCLIP?

OpenCLIP does not have a directly accessible distribution as it uses the OpenAI-CLIP model to filter data for creating the LAION dataset. Therefore implicitly it carries the OpenAI-CLIP distribution, which is unknown because OpenAI did not release it.

We would like to thank the reviewer for their insightful review. Please let us know if anything else is unclear and we will try to answer.

I find that the authors use the average accuracy across multiple datasets as a major performance metric throughout the paper.

We agree with the drawback of reporting average accuracy, but we are following this as it is standard practice in the community. We created a version of Table 8 that reports on all individual datasets and included it in the updated paper.

Some data filtering details could be further detailed. For example, the last sentence in page 4 (sorry but this submission does not have a line number) talks a bit about some noise reduction... (e.g., regular expression on dates, ids etc.).

In this part of the text we only state that sub-string matching to metadata automatically removes noisy captions (e.g. a caption just containing a date string), without using complicated hand-crafted filters (e.g. a regular expression that removes such cases).

Also, figure 5 in the appendix shows some URL & Text Dedup and NSFW Image Dedup, but does not describe them in details.

We will provide more details on our pipeline in the paper and will add the following to appendix A.2.:

• Our HTML Parser is applied to all WAT files of CommonCrawl by keeping all <img> tags with both relative and absolute URLs and one or multiple text keys.
• Language-Identification: we use an internal language identification system that can detect 191 languages with different dialects and keep all texts that are tagged as English (or its dialects).
• URL / Text deduplication: We use 24bit sha224 hashing to encode an URL and into tables to deduplicate columns with the same hash, avoiding downloading the same image URL multiple times. URLs with illegal domains are removed; if there are multiple texts associated with the same image, these are further deduplicated; texts with NSFW keywords are removed;
• NSFW filter: We use an internal system that can classify inappropriate content in images into 96 types of dangerous content and discard such image/text pairs.
• Image Deduplication: Images are further deduplicated by 64-bit PCA hash, derived from a similarity search model's feature embeddings with PCA reduction to 64 dimensions and sign quantization.

1. either a comprehensive distribution probing or visualization of the metadata used in the paper (right now only top-20 entried are showed, and they seem very general and vague wordings);

Here we further group counts of entries and show 5 examples per group as below:

This Table is now also in the appendix of the updated paper (Table 13).

The counts exhibit a long-tailed distribution which is shown in the plot of the cumulative sum of distribution in Fig 2/Fig. 3. We release the full distribution with our code.

2. remove some metadata sources (like no wordnet synset), or tune the threshold hyper-parameters like PMI or view frequency.

This is a great suggestion; however, note that ablating metadata is challenging, as this is the first step of the whole data pipeline and therefore it is difficult to backtrack quantitative signals from the downstream tasks to this step.

We conducted the suggested experiment by removing wordnet from the metadata (this results in removal of 45786 (out of 86554 wordnet entries). This removes 9M training data from the MetaCLIP-400M dataset (because wikipedia sources can cover certain wordnet entries, or one text with wikipedia matched entry can carry a wordnet entry without match). We are training on this set and will report the result later. After 6B/12.8B samples seen (iterations) the accuracy is 55.3% vs 55.6% with full 400M data scale on ImageNet.