SIEVE: Multimodal Dataset Pruning using Image-Captioning Models

We thank the reviewer for their feedback and would like to address their concerns:

"The authors argue CLIPScore has three limitations, and propose a caption-text score, where caption is generated by a captioning model on an image, and text is alt-text. However, the caption-text score could also have similar limitations"

1. We agree with the reviewer that SIEVE’s pruning signal can still result in missed hard samples and we will add a section discussing the bias of SIEVE. Nevertheless, we demonstrate that SIEVE and CLIP Score’s pruning signals are highly complementary:
   1. Compared to CLIP Score, SIEVE is more likely to focuse on image content rather than context as shown in Figure 1 of the paper. This is clear in the per-task performance on both medium and large scale where SIEVE on its own surpasses CLIP score on retrieval tasks by a large margin (+3.8%, 5.4%), while CLIP Score due to its large pretraining data, performs better on context-based tasks like identifying the geographical location from an image.
   2. We show that SIEVE when augmented to CLIPScore, beats CLIPScore on the main tasks and on Average on both medium and large scales. Hence, SIEVE is a useful filtering signal.
2. We also demonstrate in our ablation studies (table 3) that compared to CLIP and BLIP text encoders, our sentence similarity embedding space can bridge the diversity gap between the alt-text and the captioner output, improving our chances of detecting well-aligned hard samples.

"The large-scale results in Table 1 supports this - SIEVE's avg performance 52.3 is worse than CLIPScore's avg performance 52.9"

1. Our goal is not to beat CLIP Score but to provide a complementary ranking signal focused on image content aiming to reduce false positives and negatives. Fusing CLIPScore with SIEVE improves overall performance by 2.6% and 1.7% on medium and large scale respectively.
2. The reported average performance is conducted on 38 downstream tasks concealing per-task performance. For instance, SIEVE on its own performs significantly better on retrieval tasks and this performance gap increases from medium (+3.8%) to large scale (5.4%).
3. We argue that even though SIEVE is pre-trained only on 14 million image-text pairs, its performance compared to CLIPScore (pretrained on 400 million image-text pairs) is within less than 1% on the large scale, which is very promising.

"As mentioned in the paper (Sec 4.2), the results between medium and large scales are not consistent. The reviewer suspects result variance could partially explain that. How many models are trained for each filtering setting to get the quantitative results, and what is the variance?"

1. For all our ablation studies, we run 3 experiments and report the average performance in the paper. The variance of SIEVE is 0.29 on medium scale and 0.31 on large scale. For the submission on the leaderboard, we are only able to submit one pretrained model (seed=1).
2. The scale of experiments in terms of the candidate pool size, model capacity, and compute budget (maximum number of samples seen) can have a significant effect on the ranking of pruning methods. Due to the biases of each ranking method, a method that performs well on a small or medium scale does not necessarily perform better on a larger scale. This is shown both in the DataComp paper where, for instance, image-based filtering performs better than LAION filtering on medium scale but the order is reversed on large scale.
3. With respect to average performance, the order of CLIPScore and SIEVE are reversed between medium and large but the gap is within less than 1%. Nevertheless, the performance of SIEVE on retrieval tasks consistently improves from 3.8% to 5.4% from medium to large scale. Finally, fusing CLIPScore with SIEVE leads to a 2.6.7% improvement on medium and 1.7% on large over CLIPScore.

We thank the reviewer for the insightful comments and constructive criticism. We would like to address their concerns below:

"It is unclear how scalable the method is. The fact that the improvements seen at medium scale are larger than the improvements seen at large scale of DataComp are somewhat concerning for the scalability of the method. The paper would be stronger if these gains were consistent or grew as scale increased."

While the standalone performance of SIEVE is better on medium scale compared to large scale (relative to CLIPScore), the average performance for instance hides the fact that SIEVE does significantly better on retrieval tasks at scale compared to CLIPScore (3.8% on medium versus 5.4% on large). In addition, since our goal is not to replace CLIPScore but to provide a complementary ranking signal focused on image content rather than context, we demonstrate that fusing CLIPScore with SIEVE improves performances compared to CLIPScore by 2.6% on medium, while the improvement on large is at 1.7%

"While the authors present several ablations, I believe the captioning model is a central part of their pipeline. As such, I think the paper would be stronger if the authors explored more diverse captioning models."

We agree with the reviewer that exploring multiple captioning models would strengthen the paper. Since BLIP provides models pretraining on 14M curated set and 129M uncurated set, we thought it gives us a good opportunity to investigate the captioner pre-training data-distribution on SIEVE. We are currently conducting experiments using other captioning models.

We thank the reviewer for their feedback and would like to address their concerns below

"the proposed model was submitted as part of the DataComp competition, where other proposed models from other teams were submitted as well. The paper fails to acknowledge the other competitor models and does not report their results."

1. At the time of submission, T-MARS[1], an orthogonal concurrent pruning approach which relies on text masking before applying CLIPScore, was the only medium-scale submissions that was available on arxiv. We cite this approach in our literature review. We also mention in our future works section that we aim to utilize text masking in SIEVE before captioning to improve SIEVE. Finally, we conduct additional experiments showing the effect of text masking on SIEVE and report the results in Table 4 in the appendix. Works that we did not cite did not have publications associated with them at the time of submission.

2. All other methods either had no associated publication or was arvix a day before the conference deadline: “The Devil is in The Details” (DID) paper only appeared on arxiv on September 27th, one day before ICLR submission deadline and we only noticed it a couple of weeks after the deadline: https://arxiv.org/abs/2309.15954 “Data Filtering Networks” paper only appeared on arxiv on Septembet 29th, one day after ICLR submission deadline: https://arxiv.org/abs/2309.17425

3. The novelty of SIEVE is that it enables pruning image-text datasets with captioning models pretrained on small but curated datasets. “The Devil in the Details” (DID) submission focuses on fusing many ranking signals to improve accuracy, a phenomenon that was shown in the DataComp paper especially at scale. We demonstrate that SIEVE provides a promising ranking signal and demonstrate that it is indeed complementary to CLIP Score at medium and large scale. Moreover, DID aligns CLIP's pretraining data distribution with the DataComp evaluation set (downstream tasks). We argued in the paper that selecting samples that match the distribution of downstream tasks encourages overfitting to the evaluation set and, thus limits generalization to other downstream tasks. Without that distribution alignment, our approach beats DID by a large margin on average performance (32.8% versus 35.4%).

4. The Data Filtering Networks (DFN) submission, uses a CLIP model pre-trained on a non-public 357 million human curated dataset (as the paper stated that the labels were “human verified”), to rank the DataComp samples. Compared to DFN, our captioner is pretrained on 25x less data but still holds second place on large-scale. Moreover, DFN finetunes on the training set of ImageNet, the main evaluation task of DataComp. Average performance of DFN without fine-tuning on ImageNet is within 0.5% of our approach (54.6% versus 55.0%).

5. With the exception of DFN, all other methods do not report results at the large scale, while our paper shows results for both medium and large scales. The change of ordering of pruning methods is shown in the original Datacomp paper where, for example, image-based filtering performs better than LAION filtering on medium scale but worse on large-scale.

"I wonder how much the pruned model can be claimed to be trained on a 24 million dataset while, at the same time, the captioner model used to prune the model has been trained on 14 million image-text pairs and the text matching network on billions of text pairs."

Our claim is that we pretrain on 14 million image-text pairs, not 24 million. Compared to text data, curating image-text pairs is very expensive as it involves aligning two modalities. Our captioner is pre-trained on 14 Million pairs, where 12 Million samples are from CC12M where the curation process involves uni-modal and multimodal curation. On the other hand, the pretraining task for the sentence similarity model is simply to determine whether two sentences originate from the same document. Constructing such unlabelled text dataset for pretraining the sentence similarity model is trivial and does not require multimodal alignment. We use a publicly available sentence similarity model consisting of only 22 million parameters.