SIEVE: General Purpose Data Filtering System Matching GPT-4o Accuracy at 1% the Cost

Thank you for your insightful review and pointing out the unclear points of our paper. We address your concerns below.

Catastrophic forgetting: we would like to note that we are doing a standard transfer learning process here, finetuning a pretrained T5 encoder model for a specific task of data filtering. As this finetuned model is only used for data filtering and not text generation, it is actually beneficial for the model to only keep the relevant parameters for the filtering task. As an example, in a domain specific case, if we are training a filtering model for medical knowledge, the model doesn't need to remember information about graduate level math.

Unclear statements: thank you very much for pointing these out. It should be T5 encoder model throughout. And yes, the plot has the two curves switched. We will update the legends. The two rows with 'SIEVE (Ours)' both use the exact same process and only differ by the total GPT-4o query budget spent as indicated by the fourth row. The first 'SIEVE (Ours)' with a 60K budget is chosen so its accuracy matches GPT-4o, while the second 'SIEVE (Ours)' with a 25K budget is chosen so its accuracy matches the random baseline at 100K.

About imbalance ratio, we are using the randomly chosen GPT4o calls at the initialization phase (line 183) to estimate the imbalance ratio. As these GPT4o queries need to be made already, we do not incur any additional cost and is certainly feasible in practice. Note that we are only using an estimate of the imbalance ratio instead of the true imbalance ratio, and this estimate can be very accuracy with only a few hundred random samples (by tail bounds of binomial distribution).

In addition, the use of focal loss actually places the decision boundary more biased towards the majority class, which results in more unbalanced sampling during active learning. This is because, focal loss and the imbalanced classification literature in general, aims to shift the decision boundary towards where a balanced testing distribution of the two classes equalize in density. We refer the reviewer to classics like [1] for their work. In our case, we would like to shift the boundary to TRM threshold, where imbalanced unlabeled distributions of the two classes roughly equalize in density. In this case, we would like to shift the decision boundary more towards the minority class. As a concrete evidence, in this figure (https://ibb.co/z7wDK9K), we plot the ratio of unlabeled minority class examples at different sigmoid scores trained with focal loss (for the final model checkpoint of the quality filter result). A ratio of .5 means the minority and majority class examples equalize. It is clear that the decision threshold of .5 given by the focal loss is extremely biased towards the majority class. On the other hand, the TRM threshold would be where the y-axis reaches around .5, which is very different from the decision boundary given by the focal loss.

So to summarize, when labeling around .5, we will definitely label a much less balanced dataset, which hinders the model performance based on prior literature in deep active learning under pool based settings. We are also launching an ablation study that select data closest to .5, and will include the discussions in the final version of our paper.

Upper and lower bounds: these bounds are updated on line 197. This is a direct application of the empirical Bernstein bound, which ensures the TRM threshold is always within the bounds across different epochs with at least 95% probability.

In addition, the point of doing active learning is to train a new model on the latest labeled examples that are uncertain for the current model. Once the we update the current model with training on these uncertain examples, they will become well learned (certain) for the newly updated model. Therefore, the goal is exactly to labeled examples that are uncertain and balanced (within the upper and lower bounds), so that once we train on them, they become easy and certain examples for our latest model. In such cases, these examples will definitely fall out the lower and upper bounds, which is exactly what we want. The ideal ultimate goal is for there to be no snippets that fall into these lower and upper bounds (which entails 100% model accuracy).

Initial labeled set: as suggested by our notation on line 183, these snippets are sampled uniformly at random. We will add some text to make this clearer.

Finetuning pairs: due to the long prompts that we use, they do not fit into the context window of T5, so we are only training on the (snippet, annotation) pairs.

Thanks again for your review, and we would love to hear your thoughts given our response.

[1] Cao, K., Wei, C., Gaidon, A., Arechiga, N., & Ma, T. (2019). Learning imbalanced datasets with label-distribution-aware margin loss. Advances in neural information processing systems, 32.

Thank you for your insightful review and pointing out the unclear parts of our paper. We would like to address your concerns below.

1. Dataset settings: Even though we only tested on the OpenWebText dataset, different filters we tested on have induced classification tasks of different imbalance ratio (see Figure 3, which ranges from 0.026 to 0.457). In addition, as indicated by the self-consistency accuracy in Table 1, GPT-4o has noise levels anywhere from 3% to 12% across different filtering tasks. We would also like to note that the OpenWebText is a web crawl of the data on the internet, which already contains diverse topics and types of data online, including books, news articles, online forums, and more. We choose the OpenWebText dataset because of its broad coverage and its reasonable size for academic research. To our knowledge, it is hard to find a web crawl dataset of this particular scale (~10B tokens). In addition, this is the dataset GPT2 was trained, giving this dataset a lot of credibility.

2. T5 baselines: We totally agree with the reviewer we should include this as a baseline. We are running the experiment and do not expect T5 to perform well at all on this task (this can be seen by the poor performance when T5 is only finetuned on a few queries from GPT4o in Figure 4).

3. Thank you for pointing out the clarity issue when calculating the sigmoid scores. Described in Algorithm 1, our active learning algorithm iteratively trains a new neural network model based on the $B$ newly labeled samples at each outer iteration. This model $f$ is trained and denoted on line 185. The sigmoid scores are simply applying the latest lightweight model $f$ on the unlabeled snippets coming into the stream, i.e., $f(x)$ for any new snippet x. This is also detailed on line 194. We will make this more clear in our text.

4. When filtering domain specific datasets, as this is likely a subset of the web crawl already, we expect the approach taken by SIEVE to work as well. We do think leveraging domain specific foundation models could save additional queries to GPT4o than T5. We will add this discussion to the future work section.

Thanks again for your review, and we would love to hear your thoughts given our response.

Thank you for your insightful review and pointing out the unclear parts of our paper. We would like to address your concerns below.

1. The choice of $B$: Thank you for point this out. This is indeed an important parameter that deserves more documentation. The choice of $B$ is highly application and scenario dependent. Specifically, a larger $B$ means we need to retrain the model fewer number of times, thus introducing much lower training cost for the lightweight model finetuning. At the same time, a larger $B$ with less number of model updates also enjoys less of the accuracy benefit from active learning. In practice, we believe $B$ should be chosen based on the lightweight model training budget. If the lightweight model is trained on machines one owns, it costs close to nothing every time the model is retrained. In such cases, $B$ should be chosen to be smaller, so that query cost of GPT-4o can be reduced. We also think in practice the practitioners should choose $B$ adaptively, based on the validation accuracy the model obtains thus far. The practitioner would be recommended to start with a smaller $B$ (i.e. 1.5K), and switch to larger $B$ (i.e. 5K or even 20K) if the model is still far away from a target level of accuracy. The validation set can simply consists of 1K to 5K of random annotated samples.

In our experiments, we set $B$ to be $1.5K$ initially and increased it to $5K$ if the model accuracy did not achieve the target accuracy after 10 batches.

2. We fully agree with the reviewer that studying different architectures can strengthen our paper. We are launching an experiment with distillbert and will include the results and discussion in the final version of our paper. We will also try to report the results before the discussion period ends. We chose $T5$ for no apparent reasoning other than it is one of the latest encoder-based open source model.

3. Yes, the legends are flipped. We will fix that.

4. The upper and lower bounds are updated on line 197, and is a direct application of the empirical Bernstein bound. This simple ensures the TRM threshold is always within the bounds across different epochs with at least 95% probability. The bounds are dynamically and automatically adjusted for each filter and each iteration of the active learning process.

5. Applying SIEVE beyond data filtering is definitely an important future work and we will update the future work section. We would like to note that data filtering itself is a crucial problem in training LLMs today, also noted by all other reviewers. Therefore, we believe the scope of our work is addressing a sufficiently important problem already. Extending this approach to more broad application is definitely going to make a large impact, but falls out of the scope of this paper.

Thanks again for your review, and we would love to hear your thoughts given our response.

Thank you for your insightful review and pointing out the clarity issues in our paper. We respond to your concerns below:

1. Is active learning cost considered? Yes, in Table 1, lightweight model cost is the total cost of model training from active learning and inference cost on the entire dataset. In Table 2, this is specifically broken down into two separate costs of training and inference. We realize Table 1 is unclear, and we will add this detail in the paper. Thank you for pointing this out.

2. We agree that collecting human evaluation on snippets the two models agree on would enhance the results. The core message we wanted to demo here is that the lightweight model is able to achieve the same accuracy as GPT-4o. Since the accuracy is the same on snippets they agree on, we opted to spend our limited annotation budget on annotating the disagreement cases. If accepted, we will conduct additional human studies on the agreement cases as suggested by the reviewer.

3. Encoder architecture is still the most prevalent model for classification. Decoder models are more efficient from a memory standpoint (large decoder models can fit into GPUs but encoder models cannot). However, in our case, since the lightweight models are so small, they can fit into a GPU regardless of decoder or encoder architecture. In terms of total FLOPs during inference, decoder models only saves marginally over encoder models. Therefore, we decided to use an encoder based model as it attends to every token equally while only cost less than 10% of computation (in terms of FLOPs).

4. We agree that using stronger models like the multi-agent approach or even o1 can benefit the filtering quality. In such cases, the cost of GPT-4o queries will be even larger. Given the same query budget, it is an interesting question whether learning from a larger amount of noisy single GPT-4o queries would underperform a smaller amount of multi-agent GPT-4o queries. From our experience, more annotations from noisy single queries will be a more effective use of the query budget. But we agree with the reviewer this is something interesting to figure out, so we are starting an ablation study to compare the two scenarios that we will include in the final draft of the paper. Regardless the results, our system is not limited to any single teacher model (GPT4o was the state-of-art model at the time of writing), so it could definitely be adapted for more advanced teacher models as well.

5. Filtering robustness and fairness. Could we kindly ask the reviewer to elaborate what aspects of robustness and fairness you are referring to? From our perspective, robustness and fairness of any language model is impossible to guarantee. It is not clear to us which aspects of these metrics can be guaranteed for GPT-4o but not our lightweight model. We would be happy to include this in the limitation section, but are currently unclear on what specific aspects the reviewer is referring to.

6. In terms of novelty, we believe data filtering is a crucial research area as noted by the reviewer. Traditionally, data filters are trained on small quality/toxicity datasets annotated by human, leaving a large void of possibilities for domain specific and criterion specific data filters. From the perspective of data filtering methods, we believe our proposed methodology is very novel. In addition, from a technical side, the stream-based active learning algorithm that deals with imbalance and our theoretical bounds are all novel contributions to the field.

Thanks again for your review, and we would love to hear your thoughts given our response.