Data Filtering Networks

Thank you for your review. We are happy that you find our work interesting and the results impressive. We hope to address your remaining concerns.

To address your questions and concerns about HQITP we provide a copy of our general response from above. We provide some clarity about the HQITP-350M dataset and what we omitted from the paper.

• While we cannot release HQITP-350M for legal purposes, we have shown it is possible to build a DFN better than the existing OAI-CLIP models using only publicly available data. While we only did this up to the “large” DataComp scale in the original submission, we extend this to the XL scale below and compare it to the previous state of the art filtering network (OAI-CLIP-B/32). Our DFN with none of the additional modifications (fine-tuning/open-ai-init/augmentation) from section 4 in the paper outperforms the OAI DFN.
• Because HQITP-350M dataset would be similar in nature to licensed stock image datasets, so we don’t have access to the exact labeling decision behind each image caption. We also note we will release the dataset induced by our best performing DFN which is sufficient for the community to train strong models.
• In table 2 (above), We provide an additional ablation below where we control the number of examples from HQITP used to train the DFN and measure its performance on the datacomp benchmark (at the medium scale). We show that one can achieve most of the gains with a fraction of the of the full dataset. We note this experiment was done with an earlier version of HQITP that only contained 135M examples.

Each result is a single run, as these experiments are quite expensive at the highest scales. However, we can provide error bars for the results on the validation tasks - below in table 5 are 95% Clopper-Pearson confidence intervals for the results of our best model.

Thank you for your detailed review. We hope to address your concerns and are happy to answer any additional questions.

While some may not find these results surprising, we believe this is the first work that trains a CLIP model specifically for the purpose of filtering data for large-scale dataset creation, whereas all prior work relies on using an OpenAI CLIP model to filter the data.

In the paper we provided analysis for understanding DFNs beyond showing that better ImageNet CLIP models do not result in better filtering models. We demonstrated that data quality is more important for training filtering models than for training standard models. Adding model filtered data from a raw pool is a noisy process that can still include bad training points, and these points negatively affect a model used for filtering much more than it affects standard models. For additional details, see Figure 4 and Section 3.3 in the paper.

Overfitting to ImageNet/COCO/FLICKR

We understand that the reviewer is concerned about potential overfitting to ImageNet because our filtering network trains on ImageNet + Coco + Flickr (while our data pool is de-duplicated against these datasets). To address this we first provide a full results table to provide an in depth analysis of results and provide ablations at a higher compute scale that show even less signs of overfitting to ImageNet/COCO/Flickr. We also add that while fine-tuning does help performance on a set of key benchmarks, our non-finetuned DFN consistently outperforms the best and most widely used DFN today (the OAI-CLIP-ViT-B//32). We apologize for only providing averages of our evaluation which could cloud some of the takeaways. We provide a full results table (Table 3 and 4 above) for our best performing and comparable external models on the full evaluation suite. We will update our final manuscript to have links to these full results tables. We would like to highlight three points from this full results analysis

1. Performance on several of the VTAB tasks (Cifar-10, Caltech-101, STL-10) are close to saturated so making large improvements across all the tasks can be difficult, and even a 0.5% average improvement can be meaningful. We do agree that performance on a few high quality benchmarks may be more indicative of performance improvements.
2. Our benchmark contains 2 additional high quality test sets specifically for highlighting biases in traditional imagenet style models DollarStreet and GeODE, this is accounted for in the average calculation but not VTAB calculation.
3. We note the retrieval benchmark average is an average of 3 tasks (Flickr, Coco and WinoGAVIL). WinoGAVIL was a retrieval benchmark released in 2022 containing images from google images and de-duplicated from our training set.

With the in-depth results table in mind, we provide 3 additional models that provide evidence that overfitting to ImageNet/CoCO/Flickr is not a significant concern.

1. First we scaled up our ViT-H/14 method further with 1 epoch of high resolution fine-tuning and compare our results across all categories to a state of the art CLIP model that performs inference at a higher resolution: SiGLIP (particularly the ViT-SO400M-14-SigLIP variant). We show that not only does our approach out-perform SiGLIP on ImageNet (84.4 vs 83.1) and average across the 38 benchmarks (71.0 vs 69.4), our performance gap on VTAB is greater than that on ImageNet or Retrieval (68.5 vs 64.6). We show strong performance on DollarStreet and GeODE, slightly under-performing SiGLIP on the former, and over-performing on the latter. We additionally note that this is all with the SigLIP model using a more efficient architecture [2] and being trained using the same compute budget.
2. We agree our ViT-B/16 results seem to indicate we are “weakening” the model somehow by training the DFN on ImageNet, we show that at larger scale this phenomena reverses. We provide results of a ViT-L at XL datacomp-scale induced by the DFN without the fine-tuning on ImageNet/COCO/FLICKR. We see that the average performance of the model is higher for the finetuned-DFN model compared to the “base” one (that does not finetune on IN). In addition we see improvements in both WinoGAVIL (+3pp) , DollarStreet(+2pp) and a slight decrease on GeoDE (-0.7pp).
3. Finally we take our XL ViT-L model and continue training it for 3x the number of iterations (39B examples seen) and we see that while we get a slight performance win on ImageNet (+0.7pp), we get a larger gain on other aspects of our benchmark such as +1.2pp on average, +0.9pp on VTAB, +1.5pp on Geode and a slight performance degradation on DollarStreet (-0.5pp).

We thank the reviewer for their quick response. It would be helpful for us to understand the reviewer’s concerns better. For instance, the reviewer writes

I'm not confident that "better data -> better DFN" accurately describes what is happening in these experiments.

Figure 4 shows how adding even a small amount of noisy data into the DFN training set leads to worse induced models - hence “better data -> better DFN”. What aspect of this experiment does the reviewer not agree with?

Finally, we kindly ask that the reviewer re-consider their score as there now seems to be agreement on multiple positive aspects of our submission:

• The reviewer calls DFNs “an extremely valuable direction of research”
• The reviewer agrees that we have addressed the concern regarding overfitting
• Our experimental results are the best published CLIP-style models in the literature.

What remains is that the reviewer encourages us “to dive a little deeper into the relations between what kind of data the DFN is trained on and how that affects performance on specific downstream tasks.” We agree that this is an interesting direction, and plan to investigate it in future work. But in one paper it is unfortunately not possible to follow all potential follow-up questions to their end. So taken together, we wonder if the overall assessment / rating is still “3: reject, not good enough”

Thank you for your kind review. We hope to address your remaining questions.

To address your questions and concerns about HQITP we provide a copy of our general response from above:

We provide some clarity about the HQITP-350M dataset and what we omitted from the paper.

1. While we cannot release HQITP-350M for legal purposes, we have shown it is possible to build a DFN better than the existing OAI-CLIP models using only publicly available data. While we only did this up to the “large” DataComp scale in the original submission, we extend this to the XL scale below and compare it to the previous state of the art filtering network (OAI-CLIP-B/32). Our DFN with none of the additional modifications (fine-tuning/open-ai-init/augmentation) from section 4 in the paper outperforms the OAI DFN.
2. Because HQITP-350M dataset would be similar in nature to licensed stock image datasets, so we don’t have access to the exact labeling decision behind each image caption. We also note we will release the dataset induced by our best performing DFN which is sufficient for the community to train strong models.
3. In table 2 (above), We provide an additional ablation below where we control the number of examples from HQITP used to train the DFN and measure its performance on the datacomp benchmark (at the medium scale). We show that one can achieve most of the gains with a fraction of the of the full dataset. We note this experiment was done with an earlier version of HQITP that only contained 135M examples.

We use image-text similarity as a proxy for quality, but it is flawed and is not truly measuring quality. We attempted to do something similar for images with a binary filter, as well as image and text separately with a M3AE model, but found that empirically CLIP similarity score works best. See Table 1 and Section 3.3 for additional details. Nguyen et al., changes the captions, and can be combined with the methods presented in our work; however, we did not do so because generating new captions can be quite expensive, especially when compared to calculating CLIP scores.

Thank you for your additional suggestions - we have added the citations and will evaluate the additional shifts in a future revision.

Thank you for your kind review. We hope to address your remaining questions.

We agree that pointwise filtering is limited, but is computationally important due to parallelizability; however, this approach can be combined with other dataset filtering approaches, such as restricting keyword counts in the text (e.g. MetaCLIP).

While the cost of filtering is not reflected in Figure 1, DFN, DC-1B, and LAION all rely on CLIP filtering to filter the dataset. DFN and LAION use a ViT B/32 model, which is cheaper (both training and inference) than the ViT-L/14 used in DC-1B. Additionally, the DFN we train for filtering is trained for less samples seen than the OpenAI models used to filter DC-1B and LAION (5B vs 12.8B). Finally, the cost of training a ViT-B/32 (4.4 GMAC at 224px) is much smaller than that of training the best models, such as a ViT-L/14 (81.1 GMAC at 224px) and a ViT-H/14 (167.4 GMAC at 224px). So the total cost of training our best filtering model is 22e9 GMACs, while the total cost of training our ViT-L and ViT-H models are 1e12 GMACs and 6.5e12 GMACS respectively. This corresponds to a filtering overhead of 2% and 0.3% respectively. Unfortunately, we are unable to directly compare with approaches used in proprietary datasets like OAI-WIT-400M and WebLI, as their creation processes are not public knowledge.

We do plan on applying similar methods to text, as well as additional multimodal datasets, and are excited to further explore this space in dataset creation.

Table 1

Table 2