Filter Like You Test: Data-Driven Data Filtering for CLIP Pretraining

We thank the reviewer for recognizing the novelty of our method, and for acknowledging our method's strong performance compared to other publicly-available approaches. Below, we address the weaknesses and questions raised in the review.

The evaluation lacks qualitative analysis that would clarify the practical implications of the boost in quantitative performance. The paper would benefit from visual demonstrations showing how the new CLIP embeddings differ from those of the original model - for instance, through image retrieval examples or t-sne embedding visualizations.

How do your embeddings differ from the original CLIP embeddings? In what cases are your embeddings better and worse than the original CLIP embeddings?

Is there a qualitative boost in performance in addition to the quantitative gain?

We appreciate the reviewer's suggestions for qualitative analysis. We note that such embedding-level qualitative evaluation is not standard practice in the data filtering literature for CLIP pretraining. We would appreciate it if the reviewer could point to papers in the data filtering or CLIP pretraining literature that include such qualitative analysis.

As an alternative, we could provide a different form of qualitative analysis: examining which training examples our method prioritizes compared to existing approaches. This could provide insights as to which types of examples our learned scoring model values differently from other methods.

The writing suffers from slight redundancy and could be tightened. Some concepts and findings are reiterated unnecessarily throughout the paper.

The clarity and readability of the paper is very important to us. We would be grateful if the reviewer could point to specific instances where our writing is redundant or where concepts are unnecessarily reiterated. We are happy to revise our paper to improve clarity and conciseness.

Why optimize both the scoring model and CLIP reference model together? This seems like a non-trivial choice. What happens if you just train the scoring model while keeping the reference model frozen?

We agree with the reviewer that optimizing the reference CLIP model is a non-trivial choice. In Appendix C.7 we present E-FLYT ablation experiments including freezing the reference CLIP model. The results are shown in Table 11 under the "Trainable parameters" experiment, "Only scoring model" variation. The experiment shows that training the reference model yields considerably better performance than freezing it.

Intuitively, when the reference model is frozen at a single point in parameter space, the scoring model may overfit to a data weighting that is only locally beneficial for training. In contrast, when we train the reference model, the loss function for the scoring model averages over many configurations in parameter space.

We thank the reviewer for recognizing the effectiveness of our direct learning-based approach, especially compared to public-resource approaches. We address the weaknesses raised in the review below.

The Soft Cap Sampling won't eliminate the repetition of sampled data.

We do not aim to eliminate the repetition of sampled data. Our aim with Soft Cap Sampling is to carefully choose how many times each example is repeated during training.

Could you explain how the proposed Soft Cap Sampling change the distribution of sampled data?

Figure 2 shows the histogram of the sampled examples using SCS compared to the standard top 20% threshold filtering. We observe that SCS samples more unique examples - 25.7M compared to 23.9M of threshold filtering, while the highest scoring examples are sampled up to 56 times compared to 6 times of threshold filtering. This allows for more diversity in the training distribution while upsampling the best examples.

The proposed FLYT algorithm appears conceptually and technically aligned with meta-learning paradigms (...) However, the paper does not acknowledge this connection or cite relevant meta-learning literature in the introduction or related work.

In the paragraph "Learned data weighting" of the related work section, we cite [37], [43] which to our knowledge are the two meta-learning papers most relevant to our work and describe their connection with our work.

However, we agree that we should make a clearer connection between our work and meta-learning paradigms in our related work section, and will address this in the final revision. We thank the reviewer for this observation and welcome suggestions for additional especially related papers.

The porposed FLYT/Mixing-FLYT may significantly increases the training time (use second-order gradients to optimize scoring model), which is not discussed in the experiments or mentioned in the limitations section of the paper.

As noted in Section 4.1, we discuss in detail the computational cost of our method in Appendix C.2. While second-order gradients are generally expensive, our M-FLYT scoring model requires relatively few training iterations, which overall results in 3x fewer FLOPs than training a DataComp medium scale model. At larger scales (DataComp large or Xlarge), filtering cost becomes negligible relative to the overall training cost. In the revision we will mention this under limitations, though we do not consider computational cost a significant limitation.

We thank the reviewer for recognizing what we consider one of the core strengths of our work - minimal reliance on heuristics. We appreciate their acknowledgment of our method's strong performance across multiple tasks, without relying on private datasets. Below, we address the weaknesses and questions raised in the review.

Not levied at this paper alone, but the method does directly target ImageNet by using it as the downstream task, so generalization appears to rely on the generalizability of the concepts in ImageNet. This is where the concurrently proposed MGD approach, utilizing all tasks, may result in more robust vision foundation models (of which CLIP is typically considered), since it's not overfitting to ImageNet.

Our method can be easily applied using any downstream task. In fact, in Appendix C.9 we experiment with E-FLYT using 22 downstream tasks training sets (similar to MGD). Our initial experiments (Table 12) show no significant difference in performance when using all 22 tasks compared to only using ImageNet.

We chose to focus on ImageNet since it represents standard practice in the field, as the reviewer notes. More importantly, an argument is usually made in the opposite direction: that using all available training sets of the evaluation downstream tasks leads to overfitting on the evaluation sets, while only using ImageNet as a proxy demonstrates better generalization by showing that we are less dependent on the specific downstream task.

The clarity of the paper could be improved. It would perhaps be nice to read about FLYT first, to understand the algorithm, before additional complexities such as M-FLYT and E-FLYT are introduced. For example, section 3.5 could appear before 3.3 and 3.4.

We thank the reviewer for the suggestion on clarity, as it is very important to us, and an outside perspective is extremely useful. While we consider Sections 3.3, 3.4, and 3.5 to be implementation choices, we agree with the reviewer that Section 3.5 (Weighted CLIP loss) is more crucial for understanding FLYT since it introduces a new loss function. In our final revision, we will move this to section 3.3. We welcome any additional suggestions or feedback the reviewer may have regarding clarity.

Can you explain how the "Average" column in Table 1 is computed? ... Particularly, it seems as though "M-FLYT+SCS" should have the highest average, given that it beats every other method on the other categories, and yet it has a lower average score than the non-proposed methods above.

The “Average” column represents the average over all 38 downstream tasks of the DataComp benchmark. Of these 38 datasets, 16 are not represented in any of the other 4 categories. These categories follow the standard evaluation protocol for filtering methods in the DataComp benchmark:

• ImageNet: ImageNet classification only
• ImageNet distribution shifts: 6 datasets including ImageNet-V2, ImageNet-O
• VTAB (Visual Task Adaptation Benchmark): 13 datasets including Caltech101, CIFAR100
• Retrieval: 3 retrieval datasets - Flickr30k, MSCOCO, WinoGAViL

As a concrete case, on the Camelyon17 dataset, which is only part of the average calculation, our method scores 56.8% while another method (HYPE+DFN+s-CLIPLoss) scores 69.2%. That explains how our average score can be 0.1% lower despite outperforming in all other categories.

We thank the reviewer for finding our paper well-written and easy to understand, for recognizing the soundness of our meta-learning approach, and for acknowledging the effectiveness of our proposed methods. Below, we address the weaknesses raised in the review.

implementing such meta-optimization techniques is complex

We agree that implementing FLYT is not trivial, but we think that this makes our implementation (which we open source) interesting to the community.

(...) M-FLYT, which mixes individual scores using a computationally heavy and complicated meta-optimization stage

First, we note that the meta-optimization stage occurs only during the scoring model training, which we run for just 20M examples compared to the 128M used in DataComp medium scale. As detailed in Appendix C.2, the entire M-FLYT training process (the meta-optimization stage) requires 3x fewer FLOPs than training a DataComp medium scale model. At larger scales (DataComp large or Xlarge) this becomes negligible relative to the overall training cost. Moreover, our learned M-FLYT model is a linear model, which means scoring each example is very fast and computationally inexpensive.

(...) the gains in Tables 1 and 2 appear modest. M-FLYT shows relatively small improvement over "DFN-FT [our reproduction of 16]" (compared to the impact of SCS). Moreover, the performance gap between M-FLYT (…) and the simpler, training-free "IN-weighted" method (Table 2, right) is even smaller.

We invested heavily in strengthening our baselines: “ImageNet-weighted” is not a naive implementation. Specifically, the method is (as detailed in Appendix C.5):

• Standardize the per-example scores produced by each method
• Normalize the ImageNet accuracies of these methods to [0,1]
• Add $1/(r - 1)$ to these normalized accuracies to achieve a desired min to max ratio $r$
• Compute the weighted sum per example

ImageNet-weighted also requires tuning the hyper parameter $r$. It outperforms more naive baselines such as standardized sum, yet M-FLYT still outperforms it. While we agree the improvement is modest, it is nonetheless clear. When taking into account our previous point that M-FLYT is in fact not computationally expensive, as well as our next point about the built-in calibration of M-FLYT scores, we believe these improvements make M-FLYT worthwhile.

It is likely that using SCS with just the "IN-weighted" baseline (or even DFN-FT) would yield strong results

A key advantage of the FLYT training algorithm is that it naturally produces a distribution over the training examples. Other methods such as DFN-FT or IN-weighted provide scores that are not easily translated to probability distributions.

That said, we agree with the reviewer that this claim should be supported by experimental results. We ran an experiment of SCS with a repetition penalty $\alpha = 0.15$ on the scores produced by IN-weighted with $r=8$, converting the scores to a probability distribution by applying softmax (following the same approach as with M-FLYT), and present our results in the table below. We will include these results in our revised paper.

For easy comparison, the table also includes IN-weighted with $r=8$ without SCS, in addition to M-FLYT with and without SCS.

We believe these results could be improved by introducing a new scale hyperparameter where the distribution becomes softmax(scale * scores). However, this would be significantly more computationally demanding, requiring hyperparameter sweeps over both the new scale parameter and the repetition penalty of SCS.