REDUCR: Robust Data Downsampling Using Class Priority Reweighting

We thank the reviewer for their feedback and comments. We’re glad that the reviewer appreciates the importance of the problem, simplicity of our solution, and the extensive experiments that we have conducted to confirm the efficiency of REDUCR.

To address the reviewer's questions:

1. We have added a more detailed explanation of the amortized class irreducible loss approximation to the manuscript in Appendix A.3. Our method performs well across a variety of text and image datasets and our ablation studies show that the approximate model is important for REDUCR’s performance.

2. Our work builds a solid foundation for robust online batch selection and classification with a large number of classes is an important future direction that we would like to pursue (We openly recognise this in the final section of the original submission). We agree with the reviewer that training these models can sometimes be expensive even though the size of the models can be small. The problem with many classes needs to be considered very carefully and is beyond the scope of this work. Regarding the datasets: We direct the reviewer to Tables 1 and 2 in the updated manuscript in which we report strong results on two NLP datasets that have uses in other applications, for example [1] uses a model trained on the MNLI dataset to quantify uncertainty in large language models. We believe that we have enough empirical evidence to substantiate our approach.

3. We have updated Section 2 of the manuscript to further clarify this. Each datapoint x has an associated label y. Whilst c is an arbitrary class, when y == c the label of a datapoint is equal to a class c.

4. In Section 4.4 we discuss how REDUCR uses the class irreducible loss model to help identify which points are learnable. If a point has a noisy label it will likely have a high loss under the class irreducible loss model thus resulting in a low selection score and thus not being selected for the small batch b_t. The reviewer can find a detailed discussion of these ideas in Section 3 and Section 4 of [2].

5. We do not specifically consider this within our setting however, in our experiments on the Clothing1M dataset only a small clean validation dataset was available. A combination of the noisy training data and small clean validation dataset is used to train the class irreducible loss model whilst the small clean validation dataset is used to calculate the class-holdout loss term. Given our strong empirical results on Clothing1M this suggests that only a small clean validation dataset is required and can be used in the class-holdout loss term, whilst the class irreducible loss model training does not have to be a clean dataset and can use unclean data. It is practical to hand label a small validation dataset such as this and as such we did not investigate this setting further in this work.

6. We thank the author for drawing these works to our attention and have added them to our introduction and related works sections.

Our updated manuscript and the accompanying responses aim to address your observations and recommendations. Given our responses to the questions and your acknowledgment of the importance of the problem, the simplicity of our method and the extensive empirical evaluations we have conducted to confirm the effectiveness of our method, we respectfully ask that the reviewer re-consider the reject recommendation. We welcome any further thoughts and observations.

[1] Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation, (https://openreview.net/pdf?id=VD-AYtP0dve)

[2] Prioritized training on points that are learnable, worth learning, and not yet learnt (https://arxiv.org/pdf/2206.07137.pdf)

We would like to thank the reviewer for the feedback. We are glad the reviewer recognises the importance of the problem, the simplicity of our solution, and the extent of our experiments to further justify the method. In addition, we would like to direct the reviewer to our additional experimental results in the updated manuscript (see Tables 1 and 2) where we have added further results on an NLP dataset. We believe that this further strengthens our results.

The reviewer raises two points in their analysis of the paper’s weaknesses. Firstly, they argue that the Clothing1M dataset is an improper dataset for evaluation as the baselines will perform worse due to the label noise. Whilst we agree with the reviewer’s analysis that (Loshchilov et al. 2015) will be affected by the label noise, these methods still outperform RHO-Loss [2] which is specifically designed and tested on the noisy label setting (See Figure 1, Figure 3 and Section 4.3 in [2]). As such RHO-Loss is a strong baseline in the noisy label setting yet surprisingly it is outperformed by (Loshchilov et al. 2015) in terms of the worst class test accuracy (see Figure 3). Because of this, we respectfully disagree with the reviewer’s analysis that our work has unclear motivation on the basis of the inclusion of one dataset that is prevalent in the most relevant literature. REDUCR’s strong performance on a dataset with noisy labels and class imbalance only further supports the practicality of our method for real world applications.

Nevertheless, it is important to consider noisy labels carefully. We thank the reviewer for the references and have cited [3] ([a]) in the related work section of our updated manuscript due to its use of multiple models to identify and subselect non-noisy data.

Secondly, we would like to correct the reviewer’s comment that REDUCR solves the min max optimization problem at each timestep, as per Algorithm 1 we interleave gradient updates with weight updates solving the min max problem over the course of model training. In regards to a comparison of GPU run times: REDUCR consistently outperforms the random selection (uniform) baseline in terms of worst class and average test accuracy, no matter the GPU run time! REDUCR matches the best mean performance across runs of the uniform baseline in half the number of training steps. None of the REDUCR experiments had double the run time of the uniform baseline with the largest disparity being observed on the MNLI dataset where the uniform baseline took 20 hrs to complete and REDUCR took 34 hrs on the same hardware, on this dataset REDUCR matches the mean performance of the uniform baseline around 100k training steps earlier in training as seen in Figure 6. For these reasons, we are confident about the practicality of REDUCR as an algorithm.

In response to the reviewer's question: the details of our batch selection are detailed in Line 6 of the algorithm (that we clarify further; please see the updated manuscript), we select the batch b_t by selecting the individual points with the top k scores in each batch B_t. This is a common approach in the literature (see [1]) with a sampling method being another potential choice (see [2]). The relationship between examples is an interesting avenue for future work and we thank the reviewer for highlighting this, however, this is beyond the scope of the work and our empirical results show that even without this consideration, our method achieves strong empirical performance.

As we have addressed the reviewer's questions (and updated the manuscript), clarified the motivation for including the Clothing1M dataset, and addressed the practical application of our algorithm we kindly ask the reviewer to re-evaluate their score.

[1] Soren Mindermann et al.; Prioritized training on points that are learnable, worth learning, and not yet learnt. ICML (https://arxiv.org/pdf/2206.07137.pdf)

[2] Ilya Loshchilov et al.; Online batch selection for faster training of neural networks. arXiv preprint arXiv:1511.06343

[3] Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels. NeurIPS , 2018

We appreciate the reviewer for recognizing the strengths of our paper, particularly highlighting the empirical studies that demonstrate superior worst-class test accuracy, aligning with the primary objective of our paper.

• However, we respectfully disagree with the reviewer's assessment of the novelty of our paper. To substantiate our stance, we emphasize the novelty of the problem under investigation; to our knowledge, robust online batch selection has not been formally defined or researched previously. Additionally, we underscore the distinctiveness of the derived acquisition score (dependency on classes and class-holdout loss whose presence is necessary for superior performance as clearly demonstrated in Figure 4), which again differs substantially from the previous RHO-Loss. The algorithm itself is novel, relying on MW updates, leading to noticeable design differences as well as empirical performance compared to prior approaches. Having said that, our paper is novel in terms of 1) the problem formulation; 2) the class-specific selection score; and 3) the proposed algorithm.

• To clarify why this is a suitable approximation we have updated the manuscript to explain our reasoning further (please see Appendix A.3). While we tried other approximations for this term the up-weighting resulted in the most consistently stable training and we stand by our updated empirical evaluation of REDUCR in Tables 1 and 2. These highlight our strong performance on vision and text datasets. Based on our experimental results, there is no statistically significant decline, or "sacrifice," in the average performance, as indicated in Tables 1 and 2. In particular, for Clothing1M and QQP, the average test accuracy even slightly improved, potentially due to distributional shifts. Therefore, we are uncertain about the specific aspect the reviewer is alluding to. Can the reviewer please elaborate on this in case of disagreement?

• Even if there was a drop in average performance, we note that more often than not there is a trade-off between average and worst-case performance (when it comes to robust learning/optimization algorithms; as evidenced in seminal papers, such as [1] (see Table 1)). In our case, we are pleasantly surprised by the strong average performance of our algorithm (since we do not optimize for this), considering the significant gains in worst-case performance. This gain is noteworthy, and we believe it is a strength and not a weakness of our approach.

In response to the reviewer's queries, we have addressed all questions. We firmly assert that there is no evidence to justify rejecting our paper (given the provided strengths and our updated manuscript) and kindly request the reviewer to reevaluate the acceptance decision or provide additional substantiation for the provided score.

[1] Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization (https://arxiv.org/pdf/1911.08731.pdf)