Effective pruning of web-scale datasets based on complexity of concept clusters

We thank reviewer mnGt for their valuable feedback and respond to the individual concerns below.

The main idea is based on SSP-Pruning. Therefore, the technical contribution is very trivial.

We would like to stress that scaling SSP-Pruning to LAION was highly untrivial in itself because 1) it required an initial step of deduplication as well as 2) the correct choice of the encoder to calculate the image / caption embeddings. We also believe that our idea to use cluster statistics to gauge the complexity of different concepts is novel and has not been proposed for dataset pruning.

This work only reports the results on image classification. How about the zero-shot results of CLIP in retrieval tasks, such as COCO/Flickr30k image-to-text and text-to image retrieval?

We thank the reviewer for their comment and refer them to the newly added Tables 9 and 10 in the Appendix; we also included Table 10 in our general comment to all reviewers. In Table 9, we find that we outperform the best baseline released by DataComp on 35 out of 38 tasks. T-MARS have unfortunately neither released the numbers on all test sets nor the trained models which makes it impossible to compare to their results. In Table 10, we show results on the 38 tasks from DataComp for the models trained on LAION-CAT-440M, as well as on filtered versions of this dataset.

The method involves CLIP-score filtering to curate a new dataset to train CLIP. I am unsure if such involvement is fair, as the filtering itself is similar to distilling some knowledge from a well-trained CLIP. [...]Therefore, it may not be fair to claim the data efficiency for the proposed method.

We thank the reviewer for raising this important concern. CLIP-score filtering has become a standard step in almost all data selection frameworks. For example, DataComp is becoming an established benchmark for testing data pruning methods at scale, and uses CLIP-score filtering as part of most of their baselines. Thus, we are comparing methods which all use CLIP-score filtering in some form. In our Table showing DataComp results and in our Figure 1 showing results on LAION, all baselines have also used CLIP-score filtering. In that sense, our approach is more efficient with respect to other methods which also use CLIP-score filtering.

We thank reviewer orZi for their valuable feedback and respond to the individual concerns below.

We cannot determine if the current data is effective for training ViT-B-16 and ViT-L-14 models.

We have evaluated this question in our paper and would like to point the reviewer to Table 2 where we show that DBP outperforms CLIP-score filtering for different model sizes. We also include the Table below for convenience.

The majority of the experimental evaluation metric is Imagenet zero-shot top1. Is there any bias in the algorithm towards Imagenet data? Detailed results for each dataset in VTAB are not provided.

We thank the reviewer for their comment and refer them to the newly added Tables 9 and 10 in the Appendix; we also included Table 10 in our general comment to all reviewers. In Table 9, we find that we outperform the best baseline released by DataComp on 35 out of 38 tasks. T-MARS have unfortunately neither released the numbers on all test sets nor the trained models which makes it impossible to compare to their results. In Table 10, we show results on the 38 tasks from DataComp for the models trained on LAION-CAT-440M, as well as on filtered versions of this dataset.

Does the robustness of model training change when the dataset size is reduced? Please provide robustness evaluation results.

Thank you for raising this important concern. We added the following paragraph to our Results section:

The performance on retrieval and ImageNet distribution shifts is relatively lower compared to ImageNet zero-shot accuracy (Fig. 3). This trend is consistent across different baselines for retrieval tasks and we hypothesize that retrieval and ImageNet dist. shift tasks need relatively longer training (in Fig. 3 we reduce the number of training iterations/cost seen to ≤ 55.4%). To study this behavior, we measure the performance gains obtained with longer training. We increase the number of iterations for training on the 166M dataset from 41.6% (Fig. 3) to 69% and measure the difference in performance on each of the validation tasks as illustrated in Fig 4(right). We observe that among the four tasks (ImageNet, ImageNet dist. shift, VTAB, retrieval), the ImageNet dist. shift and retrieval tasks benefit the most from longer training. Therefore, we conclude that the observed performance drops on ImageNet dist. shifts and retrieval tasks can be attributed to shorter training.

The difference between DBP and SSP-Pruning is the choice of how many examples are taken from each cluster. It seems that innovation has some shortcomings.

We would like to stress that scaling SSP-Pruning to LAION was highly untrivial in itself because 1) it required an initial step of deduplication as well as 2) the correct choice of the encoder to calculate the image / caption embeddings. We also believe that our idea to use cluster statistics to gauge the complexity of different concepts is novel and has not been proposed for dataset pruning.

In Fig.3, 222M performs worse zero shot accuracy than 166M, please give a reasonable explanation.

Let’s recall that data quality is different from data quantity. In this work, we reduce the dataset size while maintaining and/or improving the quality of the data by balancing the data clusters and removing easy examples. This increases the marginal information gain the model gets from every training batch. We pruning a dataset we need to balance between the dataset size and the dataset quality. The 222M dataset (lower quality) and the 166M (higher quality) are good examples of this balance for the ImageNet classification task. We notice that same performance behavior when we look at the LAION-440M (lower quality) dataset and the filtered datasets (higher quality) in Fig 3.

Some previous works like, [1] show theoretically and practically that balancing the data by “removing” examples from the majority groups/classes can result in a better worst group/ class performance and a better model even though the dataset size gets reduced. We added a discussion on this point to our Results section.

[1] Arjovsky et al. “Throwing away data improves worst-class error in imbalanced classification”, ICML 2023

Why do we use different architectures for filtering on LAION (CLIP-B/16) vs DataComp (CLIP-L/14)?

The reason we use two CLIP models for computing the CLIP score (B/16 for LAION and L/14 for Datacomp) is to comply with the baselines used in the literature by using the meta-data clip score for each dataset. For example, SemDeDup uses CLIP-B/16 to deduplicate LAION; also, CLIP-B/16 was used to filter LAION-440M. On the other hand, DataComp chose to use CLIP-L/14 for filtering which forced us to also use this model to remain comparable to their reported baselines.

It is also important to note that for a given dataset (LAION or Datacomp) we use the same CLIP filtering model for all the datasets we created and that keeps the comparison between models fair and consistent for each dataset.

The results section is unclear due to inconsistencies in the choice of the encoder / training epochs etc.

We thank the reviewer for this important feedback. We restructured the Results section and separated the analysis experiments on the LAION dataset (main text) and the analysis experiments on how to prune the DataComp dataset (now in Appendix F) into two sections.

We thank the reviewer for providing this valuable feedback. Consequently, we have restructured the results section, segregating the analysis experiments conducted on the LAION dataset (included in the main text) from the analysis experiments on how to prune the DataComp dataset, which are now found in Appendix F.

The new structure is as follows:

• Fig.1, Fig. 3 and Table 3: FIltering the LAION-440M
• Table 1: Filtering the DataComp dataset
• All the remaining figures: Method analysis and hyperparameters tuning using a small dataset (LAION50M).

We are happy to discuss and accommodate further requested changes with reviewer u5qD.

We thank reviewer u5qD for their valuable feedback and respond to the individual concerns below.

Is using the QP solver necessary?

Using the QP solver provides a minimal overhead–it requires between 5 and 15 minutes on a single CPU depending on the dataset size. We agree that the relatively simple program takes an excessive amount of text in the paper; thus, we have moved the optimization problem and the algorithm to the Appendix.

Please provide a broader ablation on the number of clusters.

We thank the reviewer for this suggestion and ran an experiment for a more fine-grained analysis on the number of clusters. To save compute, we trained for one epoch on our 166M dataset (Fig. 3) for a wider range of values for the number of clusters hyperparameter (50, 100, 500, 1000, 5000, 10000, 20000, 60000, 10000).

Why do you not concatenate CLIP's text + image embedding?

We briefly tried this during the development stage of the project and found that it doesn’t work better than using the multi-modal embedding. We think that the BLIP embedding shares information between the image and caption embeddings which is missing in a simple concatenation.

Why does SemDeDup do worse than LAION-440M on VTAB, but DBP does better? Analogously, why does DBP (166M) do better than SemDeDup (280M) or LAION-440M?

Thank you for raising this concern. We added the following paragraph to our Results section: A smaller, more balanced dataset can lead to better models (Fig. 3). In this work, we reduce the dataset size while maintaining and/or improving the quality of the data by balancing the data clusters and removing easy examples. This increases the marginal information gain the model gets from every training batch. As a result, we observe better performance on a variety of distribution shift tasks with shorter training: The model trained on the SemDeDup+DBP-222M dataset matches or outperforms training on the full LAION-CAT-440M dataset in all categories in Fig. 3, despite using only half the compute. The work of Arjovsky et al. (2023) also shows theoretically and practically that balancing the data by “removing” examples from the majority groups/classes can result in a better worst group/class performance and a better model even though the dataset size is reduced.

Arjovsky et al. “Throwing away data improves worst-class error in imbalanced classification”, ICML 2023

What are DBP’s shortcomings compared to T-MARS?

Unfortunately, the T-MARS authors have not released the numbers on the individual distribution shifts, making it impossible to perform a fine-grained comparative analysis of both methods.

Why do we see performance drops on ImageNet dist. shifts and retrieval tasks?

Performance on retrieval tasks is relatively lower than on the other tasks. As this trend is consistent across different baselines, we hypothesize that retrieval tasks need relatively longer training (In Fig. 3 we reduced the number of training iterations/cost seen to <=55.4%).

To study this behavior we conducted the following experiment: We compare the relative gains from longer training on the different tasks in Fig.3. In more detail, we compare the difference in performance gain when training for 41.6% or for 69% of the number of iterations compared to training on the full dataset. We show the difference in performance on each of the validation tasks in the table below. We observe that among the four tasks (ImageNet, ImageNet dist. shift, VTAB, retrieval), the ImageNet dist. shift and retrieval tasks benefit the most from longer training. Therefore, we conclude that the observed performance drops on ImageNet dist. shifts and retrieval tasks can be attributed to shorter training.

We added this discussion and the Table (as a bar plot) to the Results section of our revised paper.

We thank reviewer 7m8Z for their valuable feedback and respond to the individual concerns below.

Missing baselines for comparison, including random selection and other data pruning methods [3-4].

We thank the reviewer for their suggestions.

• Random selection: We excluded the random pruning baseline to save compute because it has been proven to be very weak. For example, [1] shows that random pruning consistently lags behind the SemDeDup baseline across various dataset sizes and evaluation tasks. Random selection has been tested by the DataComp creators on the “small” scale where it underperformed all other baselines (Table 12, https://arxiv.org/pdf/2304.14108.pdf). [1] Abbas et.al. 2023 https://arxiv.org/abs/2303.09540
• Comparison to [3]:
  • The task considered in [3] is not compatible with ours. In [3], the authors tackle the problem of semi-supervised learning and select the coreset of the unlabeled data resulting in minimum labeled set loss when trained upon in a semi-supervised manner. In contrast, we consider the fully unsupervised setting where we train a CLIP model on a pre-chosen coreset.
  • Further, The dataset and model scales of [3] differ strongly from ours. In [3], the authors perform experiments on small-scale datasets such as CIFAR10 and SVHN using WideResNets and LeNet architectures, while we perform experiments on web-scale datasets using transformer architectures. Sorscher et al. showed that methods which perform well on small-scale datasets do not scale to ImageNet, and we found scaling SSP-Prototypes up to LAION / DataComp highly untrivial. We hypothesize that scaling [3] to DataComp to enable a fair comparison will be highly involved, as well as it is not clear that the small-scale datasets are sufficiently large to train transformer models on.
• Missing comparison to [4]: While the setting of [4] is comparable to ours since the authors of [4] also study CLIP training on large-scale datasets, their choice of datasets is different from ours. Our base datasets are either LAION-CAT-440M (which is a filtered version of LAION-2B) or DataComp Medium, while [4] conduct CLIP experiments on CC3M. To enable a fair comparison, we would either need to scale [4] to DataComp or implement our method on CC3M; neither of these options is trivial and cannot be done during the short rebuttal period. If the reviewer would like to see results of DBP on CC3M, we would be happy to run these experiments until the camera ready version.

The description in the methods section is confusing.

We rewrote and restructured the Methods section to improve clarity. In particular, we rewrote the paragraphs on deduplication and CLIP-score filtering. Following the suggestion of reviewer u5qD, we moved details on the QP program to the Appendix.

The related work section contains excessive content, duplication and there are missing works which should be discussed.

We thank the reviewer for their suggestions and apologise for missing these important papers. We added them to our related work section which now has a new paragraph titled “Data curation in supervised learning” where we included several other works in addition to the reviewer’s suggestions. We also removed redundancy between Introduction and Related Work.

The tasks are relatively simple (ImageNet zero-shot). Could we compare DBP with other baselines on different tasks?

We thank the reviewer for their comment and refer them to the newly added Tables 9 and 10 in the Appendix; we included Table 10 in our general comment to all reviewers. In Table 9, we find that we outperform the best baseline released by DataComp on 35 out of 38 tasks. T-MARS have unfortunately neither released the numbers on all test sets nor the trained models which makes it impossible to compare to their results. In Table 10, we show results on the 38 tasks from DataComp for the models trained on LAION-CAT-440M, as well as on filtered versions of this dataset.

Thank you for this interesting suggestion. There would be two ways to implement DBSCAN as a baseline: (1) as a replacement of DBP which is based on k-means clustering or (2) as an additional step before performing kmeans clustering to potentially remove outliers from the data. In the following, we will comment on both.

Considering option (1):

We could use DBSCAN instead of DBP which is based on k-means clustering. We could sample data points based on the density which we could estimate as the mean distance between cluster members. We see the following issues with this approach:

• Clusters obtained with DBSCAN do not offer a simple way to rank cluster examples. Our method (as well as other coreset selection methods, see our revised Related Work section) rely on some ranking metric to select examples to train the model on and most methods propose to take the most difficult examples. K-means has the advantage of spherically-shaped and prototype-based clusters which allows us to rank the examples by measuring their distance from the cluster centroid. DBSCAN does not provide us with cluster prototypes which we could use to rank examples. While we could define cluster centroids as the mean or median of all cluster members, we are doubtful how accurate or useful these estimates would be, given that the clusters could have any (potentially non-convex) shape. For example, the means of two concentric circular clusters would coincide and lie outside of the actual clusters. Nevertheless, we could test DBSCAN pruning by taking a random subset of each cluster; however, in our early experiments, we observed that randomly selecting data points using our density-based criterion strongly underperformed compared to taking the most difficult examples.
• K-means runs on GPU using $faiss$ and is a lot faster than DBSCAN: The time complexity of k-means clustering is $O(nki)$ (note that $k$ (num. clusters) and $i$ (num. iterations) << $n$) while DBSCAN can have a worst-case time complexity of $O(n^2)$. For example, based on our experiments, $k * i$ can be around 100,000 while $n$ can be >100,000,000 (1000 orders of magnitude higher). We are uncertain how feasible DBSCAN would be for our large-scale datasets containing millions of training images. The $faiss$ implementation of k-means takes around 1-4 hours depending on the dataset size; a 1000-fold increase in clustering time would not be feasible for our dataset scales.

Considering option (2):

We could use DBSCAN to filter out outliers from the data before clustering. This might stabilize our kmeans algorithm, but would also introduce two new hyperparameters (the maximum distance between two points and the minimum number of samples in a cluster). An additional issue is the massively increased time complexity of the whole method (see the point above).

Table: Results on 38 tasks from the DataComp evaluation suite when training on the filtered LAION-440M dataset. See also Tables 10 and 9 in the paper.