[Limitations, Q2]. Pair construction strategy is interesting but not well motivated by the noisy label topic casting concerns regarding contribution. The maximally contrasting pairs of fragments ignores order relation.

We would like to clarify that ConFrag is strongly motivated by the topic of noisy label regression.

Focus on noisy label regression. A method tailored for noisy regression problems must primarily focus on detecting noise with high severity, rather than treating all noise as equal. The components of our method such as maximally contrasting fragment pairing and fragment prior, which are based on the distance in continuous and ordered label space, are designed to prioritize filtering out such high severity noise.

Order Relation. Our framework leverages ordinal relations by training on contrastive fragmented pairs to learn robust features. These features are then aggregated and ordered. Clean sample selection is done through the mixture of neighboring fragments which ensures the integrity of the learned order relations.

W2. Discuss or compare baselines considering ordinality since binary classification ignores ordinality which however was thought important in previous regression works + Limitations state that “the connection with recent work lacks deep discussion.”.

We discuss additional works that consider order information in Appendix E.1. Furthermore, we evaluate noisy regression baselines that account for ordinality in the introduction (ln 35-42), specifically C-Mixup [1] and OrdRegr [10]. OrdRegr is a noise transition matrix based loss correction method which requires noise rate estimation. Even when provided with ground truth noise in synthetic settings, the algorithm was highly ineffective. For example, in IMDB symmetric 40% noise experiments, OrdRegr performed at least 31.86% worse than other baselines in terms of MRAE. Therefore although we have implemented both methods, we only report C-Mixup results. We will ensure that OrdRegr results are included in the manuscript as well.

To the best of our knowledge, the most recent published related work is "Robust Classification via Regression for Learning with Noisy Labels." which improves the classification by reformulating it as a regression problem. We will include this to references. If there are any other works we have overlooked, kindly let us know and we will make sure to review and include it.

Additionally, we would like to reemphasize that our framework utilizes ordinal relations by training binary classifiers on contrastive fragmented pairs to develop robust features. These features are then aggregated and ordered. Clean sample selection is performed through the mixture of neighboring fragments, ensuring the preservation of the learned order relations.

Q1. You pointed out the advantages could be better generalization and robustness. Regarding the generalization, were you referring to its performance on clean-label data training? Then is Fig. 2(c) on noisy or clean data?

Generalization in the motivation of contrastive fragment pairing refers to the generalization of feature extractors trained on noisy datasets, as obtaining robust and generalizable features is crucial for sample selection in noisy label settings. Fig. 2(c) shows that under symmetric 40% label noise, training expert feature extractors on contrastive fragment pairs is better for sample selection (and thus better regression performance) than training a single feature extractor on all fragment because they are less prone to overfitting and learn more generalizable features. We will also make it clear that the results in Fig. 2(c) are on noisy data.

Q3. Is the proposed method noisy ratio unaware and applicable to any level of noisy ratio?

The reviewer correctly noted that knowing the ratio of noisy labels beforehand is beneficial but challenging to estimate in practice. We address this point at the start of Section 2, line 86, by stating, 'ConFrag is noise rate-agnostic unlike prior methods as it operates without knowing a pre-defined noise rate.' This highlights that our method does not require prior knowledge of the noise ratio.

Q4. Clean samples selection strategy from line 185-188 looks intuitive. You can certainly consider two views somehow complementary when necessary, but your methodology did not touch the representation learning.

We use the term 'representation' to indicate that our fragment pair trained binary classifiers automatically learn data representations. This means they can be considered representational learners or feature extractors. However, we acknowledge that 'representation learning' often refers to techniques like self-supervised learning. We will clarify this distinction in the manuscript.

Q5. Zhang et al., 2023 in line 91 on page 3 seems a strong baseline, which compute distance on representation. May I know why it is not included?

OrdinalEntropy [8] proposes a regularizer that learns higher entropy features by increasing the distance between representations while preserving the ordinality via weighting the representation and target space distances. Since it doesn’t tackle the noisy regression problem directly on its own, it was not included as a baseline in the manuscript. However, it is surely something we can try to mix together with other baselines as well as our method to enhance the ordinality and to better learn the high-entropy features. Results in Table R3 show that combining OrdinalEntropy with vanilla, baselines, and our method shows a slight drop in performance.

W1. some unclear illustrations

we will enhance Figure 2(a) to aid the understanding. If there are any other illustrations that require attention we would be happy to revise!

W3. clearly state technical contributions

We will highlight the technical contributions and novelty better in the manuscript, especially within the introduction.

W1. The motivation behind the idea should be stated more clearly in the introduction section. Since there is a close connection between labels and features, it is necessary to clarify why contrastive fragment pairing is introduced and whether the design of the data selection metric considers the connection.

We appreciate the feedback and suggestions! To clarify the motivation behind our idea, we will elaborate on the rationale for contrastive fragment pairing. Our method addresses the challenges of learning from noisy data by employing distinctive feature matching, which improves generalization [12,13]. Additionally, this approach helps convert closed-set noise into open-set noise, which is less harmful to the learning process [14]. We will also explain that our data selection approach, the mixture of neighboring fragments (Section 2.3), considers the correlation between labels and features that the samples with close label distances are likely to have similar features. This correlation is essential for effective fragmentation and sample selection using the mixture of neighboring fragments.

W2. Certain statements in the paper require further clarity. For instance, the meaning of "f" in "all of the noisy labeled data (f…)" needs explicit clarification upon first mention, although the meaning of the symbol can be inferred in subsequent discussions. Additionally, the phrase "we employ neighboring relations in fragments to leverage the collective information learned" in the introduction warrants elaboration.

We will make sure to include that $f$ means the index of the fragment that the data is assigned to upon the first mention! We will also elaborate the phrase "we employ neighboring relations in fragments to leverage the collective information learned," explaining that it utilizes a mixture model [15] to achieve probabilistic consensus in both prediction and representation spaces.

W3. The performance on existing noisy regression datasets?

We appreciate your recognition of our four curated benchmarks, covering age, price, and music production year prediction tasks. As suggested by the reviewer, we further evaluate the performance of our approach and baselines on two noisy regression datasets from existing noisy regression literature. The first dataset is from SuperLoss [9, 11], which injects 0.2, 0.4, 0.6, and 0.8 symmetric noise into the UTKFace dataset. The second dataset is the IMDB-org-B dataset, a real-world noise dataset studied in [16, 3, 17]. The results can be found in Table R4. We perform on par with or better than the baselines on both datasets as shown in the table below.

Q1. Could the nature of closed-set noise transformation into open-set noise be used advantageously during the data selection metric?

Thank you for the insightful question. As illustrated in Fig. 2(a), transforming closed-set noise into open-set noise can also be seen as converting it into anomalies or outliers. By leveraging the extensive literature on anomaly and outlier detection techniques, we could effectively handle open-set noise to enhance the data selection process.

W1. Focus on regression is limiting. ConFrag transforms regression tasks into classification tasks; it could potentially be extended to address noise in classification

We believe that noisy regression is an important task on its own! However, most noisy label learning research focuses on classification tasks. Regression assumes a continuously ordered relationship between features and labels, a premise well-supported in existing research [5, 6, 1, 7], whereas classification is categorical rather than ordered. Our experiments demonstrate that many noisy classification approaches do not perform well in noisy regression tasks mainly due to the fundamental difference between regression and classification. We employ cross-entropy loss (classification) as a surrogate objective to enhance the robust feature extraction, which in turn improves sample selection used to solve the regression task. This approach has been both theoretically and empirically validated thanks to better feature learning [8], as detailed in Appendix D.1. Table 2 compares regression-trained experts (ConFrag-R) and classification-trained experts (ConFrag), showing the superiority of using surrogate classification objectives. The potential for extending our approach to classification exists, but it is out-of-scope of this work.

W2. Detected noise analysis, specifically, close-set noise detection and noise in the center or boundaries of the fragments.

Close-set noise detection. Fig R1 shows the selection rate of closed-set noise on the IMDB-Clean-B dataset with symmetric 40% noise. As training progresses, they become less likely to be selected, showing ConFrag’s ability to detect closed-set noise.

Boundary, Centre noise. We compare the selection rate and the error reduction rate (ERR) between the samples at the boundary and center of fragments. Table R1 shows that the average difference of the selection rates and ERR between two groups is 2.29% and 2.43%, respectively. These results confirm that ConFrag consistently performs robust sample selection regardless of the sample's position.

[W3,4] Do additional ablation on the amount and strength of noise (with emphasis on 1%, 0.5%, 5% noise) and also evaluate on real-life noise. unclear whether increasing the number of fragments would be beneficial in extreme small noise.

We perform thorough experiments on the diverse amount and strength of noise following previous research on noisy regression [1, 9, 10, 11]. It involves the ablation on symmetric noise levels of 20/40/60/80% and Gaussian noise with a standard deviation of 30/50%. Notably, we are the only work that tests on both symmetric and Gaussian noise! The results presented in Section 4.3 show that ConFrag is effective across all of these noise ratios and strengths.

In Table R2, we analyze ConFrag using tiny noise amounts (0.5/1/5% symmetric noise) and strengths (Gaussian 2/5%) on IMDB-Clean-B.

For noise ratios of 0.5/1/5%, ConFrag achieves a very low ERR, indicating that it can detect noisy samples regardless of the noise ratio. However, when the noise strength is very small (Gaussian 2/5%), ConFrag is not able to filter them effectively. We acknowledge this as a limitation of our method. Since ConFrag primarily focuses on sample selection, it can be easily reinforced by other techniques during the training process to mitigate the effects of tiny noise strengths. Indeed, in the Gaussian 2/5% experiments, integrating C-Mixup or Co-teaching with our ConFrag improves the performance by 4.74% and 4.51%, respectively. Regarding the reviewer’s question on whether increasing the number of fragments can be beneficial when the noise strength is small, please refer to our answer to Q1. Last but most importantly, we evaluate the performance of ConFrag on real-world noise, which includes many types of noise with varying strengths. Table R4 reports the results on a version of the IMDB dataset with real-world noise [3], IMDB-org-B. The results show that the vanilla version of our method performs on par with other best-performing baselines on real-world datasets. Importantly, since ConFrag only concerns the sample selection process, it can be easily integrated with other noisy label methods. By combining our approach with other techniques, ConFrag outperforms all baselines by a non-trivial margin. Demonstrating the practical effectiveness of our approach.

Q1. A systematic way to determine the fragment number?

Analysis in Appendix G.2 shows that fixing the fragment number $F$ to 4 yields the best performance for the IMDB-Clean-B and SHIFT15M-B datasets, as it is a non-sensitive hyperparameter to the performance. This setting was subsequently used in all our experiments. The analysis further shows that given a large enough dataset, the sensitivity to the number of fragments decreases.

Finer fragmentation can improve detection of tiny strength noise but increases the risk of overfitting due to fewer samples per task. In larger datasets, this risk is mitigated because the data size per fragment remains large enough, allowing ConFrag to benefit from finer fragmentation. Based on these insights, we recommend starting with $F=4$ and incrementally increasing it until performance deteriorates, especially for larger datasets.

Q2. Limitations to the statement, "If x is more likely to belong to fragment 2 than 5, then it should be more likely to belong to 1 than 4 and 3 than 6."? Can it hold without this relationship?

The statement is highly related to the fundamental characteristic of regression: the continuous and ordered correlation between the label and feature space [5,6,1,7]. Since our approach is rooted in this characteristic, its effectiveness can be compromised if it does not hold. However, we reemphasize that this is a common characteristic found in most regression tasks, and prior works on regression with imperfect data build their methods on top of this characteristic [5,6,1].

W. Paper is overwhelmed with its presentation of too many details, which can move to the appendix to let up the reader focus on the important parts.

We sincerely thank the reviewer for the suggestion. In order to balance the clarity and the detail, we will move some overly detailed procedure of contrastive fragment pairing in Section 2.1 (line 107-116) to Appendix, and make the motivation part (line 117-144) more concise. To improve clarity better, we will include a figure illustrating the overall framework of ConFrag, highlighting the online nature of the filtering and training processes.

Q1. Extension to generative tasks

ConFrag learns to select samples $(x, y)$ that are better aligned and use them for training regression tasks (i.e., learning $P(y|x)$). As the reviewer suggested, the selected samples can also be used for conditional generation tasks (i.e., learning $P(x|y)$).

To verify this, we train a continuous conditional GAN model (SVDL+ILI) [4] for 40k steps using (1) clean IMDB-Clean-B dataset, (2) IMDB-Clean-B with 40% symmetric noise, and (3) samples selected by ConFrag on the noisy dataset. To measure the condition alignment and the quality of generated samples, we use MAE and intra-fragment FID on 52,000 generated images. Specifically, intra-fragment FID is the average of FID values measured for images in each fragment. We use $F=4$ as done in ConFrag. For both evaluation metrics, lower is better. The results show that sample selection by ConFrag improves both the condition alignment and the quality of conditionally generated images.

Q2. Extension to classification tasks

As suggested by the reviewer, it is possible to apply our contrastive fragmentation approach to classification tasks by making the following adjustments:

1. We set each class as a fragment (i.e., the number of fragments $F$ = the number of classes)
2. We measure the distance between fragments/classes. Possible metrics include euclidean distance between the GloVe embedding, CLIP text cosine similarity of each class, or CLIP image similarity using the samples of each class. The distance metric is used for constructing contrastive pairs and defining two neighboring fragments for each fragment.
3. We redefine the fragment prior (Equation 2) using the distance between classes.

We evaluated the extended ConFrag on CIFAR-10 classification with 20/40/60% symmetric noise ratios using selection ratio and precision as metrics, where precision is defined as P(clean|selected). We found that samples selected by the modified ConFrag are cleaner than the original dataset (i.e., precision higher than 1 - noise ratio), showing the potential of extending ConFrag to classification tasks.

However, since ConFrag is designed to leverage the characteristics of noisy regression tasks as mentioned in the general response, and thus noisy classification is out of our main focus, its performance does not reach the level of state-of-the-art sample selection approaches designed for noisy classification tasks. Additionally, we found that the choice of the distance metric has a significant effect on sample selection performance, indicating that the design of a better distance metric is necessary for successful extension of our approach to noisy classification tasks.