Stochastic Order Learning: An Approach to Rank Estimation Using Noisy Data

Thank you for your constructive comments. Please find our responses below.

---

Rank estimation vs. order learning

Rank estimation is a "task" that aims to predict the ordered class of an object. On the other hand, as stated in L106, order learning is an approach to rank estimation problem. Specifically, order learning algorithms exploit ordering relationship between instances for rank estimation.

Gaussian noise

• As stated in L137, the noise is assumed to have a "discrete" Gaussian distribution. Therefore, the observed rank $r_x$ in Eq.(1) will still be integers after the Gaussian noise is added.

• In practice, when annotators make errors in ranking, they are more likely to choose neighboring ranks than far-away ranks. As stated in L140-142, the symmetric($p_s=p_{-s}$), and unimodal($p_s \geq p_t$ for $0\leq s \leq t$) property of a Gaussian noise distribution well-models this trait. Thus, we chose the noise to follow a Gaussian distribution.

• As stated in L136, $e_x$ is the label error of an instance $x$ and $\mathbf{e}$ is the random variable underlying each error $e_x$. We are not sure of what exactly the point of the question here is. We will do our best to answer your question if you could specify it.

Motivation

Our motivation was to make a noise-robust algorithm suitable for rank estimation. Although many noise-robust classification methods have been developed, they yield poor results when applied to rank estimation. Most rank estimation methods, on the other hand, do not consider label noise. Specifically, the state-of-the-art method[1] designs an embedding space that arranges instances according to deterministic ranks. Using deterministic ranks fails to consider the possibility of rank label errors. In order to handle noisy data, we were motivated to use stochastic variables to model the label errors. We derived an embedding space in which instances are arranged based on probabilistically associated ranks instead of deterministic ranks.

[1] Geometric order learning for rank estimation. In NIPS, 2022.

---

If you have any additional concerns, please let us know. We will address them faithfully. Thank you again for your constructive comments.

Thank you for your constructive review and valuable suggestions. We have revised the paper to address your comments faithfully, and highlighted the revised parts in blue. Please find our responses below.

---

Novelty

SOL is the first order-learning algorithm to handle label noise by designing a stochastic approach. GOL[1] designs an embedding space that arranges instances according to their ranks. However, the ranks assigned are deterministic, failing to take label noise into account. To model label errors, we use stochastic variables and we develop the discriminative loss and the stochastic order loss to derive an embedding space in which instances are arranged based on probabilistically associated ranks. Furthermore, we propose outlier detection and relabeling schemes to reduce the overall noise level of a given dataset. Experiments on various datasets prove that our algorithm shows the best performance for rank estimation tasks under label noise.

[1] Geometric order learning for rank estimation. In NIPS, 2022.

More comparisons

As you suggested, we compare our algorithms with more comparative methods[1,2,3] including recent benchmarks. Below lists the comparisons on the MORPH II dataset. The proposed SOL still achieves the best scores. Recent noise robust-classification methods[1,2] cannot distinguish between subtle differences across ordered ranks, thus they yield poor performances. The results for other three datasets(CLAP2015, AADB, RSNA) are similar as well. We included and discussed these comparisons in the revised paper. Please see Table 1 on page 6 and Tables 12, 13, 14 on page 16 and 17.

[1] Scalable penalized regression for noise detection in learning with noisy labels. In CVPR, 2022.

[2] Active negative loss functions for learning with noisy labels. In NIPS, 2023.

[3] C-Mixup: Improving generalization in regression. In NIPS, 2022.

Dataset section

We agree with you that moving the dataset section to the Appendix adds more space for experimental results. Following your suggestion, we moved the detailed dataset descriptions to Appendix E on page 16 of the revised paper. Thank you for your constructive comment.

---

We have revised our paper to address your comments faithfully. We hope that this revision resolves your concerns. If you have any additional comments, please let us know.

Thank you for your constructive review. We have revised the paper to address your comments faithfully, and highlighted the revised parts in blue. Please find our responses below.

---

Comparison to classification methods

Unlike classification tasks, data in rank estimation have "ordered" classes. In other words, labels of ordered data are continuous, thus the difference between classes can be very small. As stated in L41-42, classification methods suffer from discriminating subtle differences across ordered classes. The table below compares the proposed algorithm to recent classification methods on the MORPH II dataset. The proposed SOL significantly outperforms the classification methods in noisy rank estimation tasks. We have included these comparisons in the revised paper. Please see Table 1 on page 6 and Tables 12, 13, 14 on page 16 and 17.

[1] Scalable penalized regression for noise detection in learning with noisy labels. In CVPR, 2022.

[2] Active negative loss functions for learning with noisy labels. In NIPS, 2023.

Writing

With all due respect, we do not agree with the reviewer. Please provide specific examples. We will do our best to revise the paper.

Definition of $d(\cdot,\cdot)$

In L156, $d(\cdot, \cdot)$ is defined. It denotes the Euclidean distance between two points in the embedding space.

Noise range

In real-world problems, generally, the range of true labels is unknown. Hence, we do not impose constraints to ensure that $r_x$, the rank given after noise $e_x$ is added, is within the range of true labels in Eq.(1). However, we clip the values so that $r_x \geq 0$, to avoid impractical (i.e. negative) ranks.

Relationship of $\mu_{r_x+s}$ and $p_s$

If the label noise of instance $x$ is $s$, the true rank of $x$ would be equal to $r_x+s$. Since $p_s$ denotes the probability that the label error of $x$ is $s$, $x$ is associated with $\mu_{r_x+s}$ with probability $p_s$. If we have misunderstood your comment on clarifying the relationship between $\mu_{r_x+s}$ and $p_s$, please let us know and we will address them more faithfully.

Equation (5)

Yes, all data is scanned when computing the centroids in Eq.(5).

Probability in Equation (2)

As stated in L137, the random variable $\mathbf{e}$ is assumed to have a discrete Gaussian distribution. Thus, the probability in Eq.(2) is assumed.

---

If you have any additional concerns, please let us know. We will address them faithfully. Thank you again for your feedback.

Thank you for your feedback. We appreciate it greatly.

---

Difference from classification

Rank estimation (or ordinal regression), which order learning algorithms aim to solve, is fundamentally different from ordinary classification. Unlike classification, different errors have different severities in rank estimation; misclassifying a 20-year-old as a 10-year-old is severer than mistaking it for a 22-year-old. Thus, for reliable rank estimation, it is important to avoid such severe errors. However, classification-based approaches do not consider this ordinal property in rank estimation datasets, and thus yield poor performances when applied to noisy rank estimation tasks. We clarified this by adding more explanation in the introduction of the revised paper.

Also, the training method of SOL is different from that of classification methods such as SPR and ACL. While they employ cross-entropy based losses which equally treat all noise (or label deviation), SOL exploits the discriminative loss and the stochastic order loss to leverage the ordinal relationship of instances/classes. Experimental results including Table 1, 12, 13, and 14 show that SOL outperforms classification-based approaches by meaningful margins.

Efficiency

Below, we measure the time required to compute the centroids in Eq.(5). We use an RTX 4090 GPU. Even for the RSNA dataset which consists of 12,611 training samples, it takes only a few minutes. It is fast enough for most use cases, since centroid update is performed only once at each epoch. Thus, as shown in the table below, the centroid update accounts for only a small portion of the overall training time. Moreover, even with very large datasets, we can easily reduce the computation time by subsampling the training instances for the centroid update.

For large scale training, memory efficiency is also important. We compare the number of parameters of SOL with that of conventional methods. SOL requires the smallest number of parameters, showing its potential for large scale applications. We have included these results in the revised paper. Please see Tables 17 and 18 in the Appendix.

$\sigma$ in Eq.(2)

In practice, prior knowledge of noise is not given. To simulate such practical settings, we also assume that noise information is unknown. Hence, $\sigma$ in Eq. 2 is a hyper-parameter in SOL and determined empirically. We have already provided the analysis of $\sigma$ in Table 9 (also in the table below). In general, SOL provides decent results at $\sigma = 1$ regardless of ground-truth noise variance. Therefore, we use $\sigma=1$ as our default option for all experiments.

---

If you have any additional concerns, please let us know. We will do our best to resolve them.

Thank you for your positive review and insightful suggestions. We have revised the paper to address your comments faithfully, and highlighted the revised parts in blue. Please find our responses below.

---

Formulas (12)~(14)

As you pointed out, the difference between the true labels of $x$ and $y$ is $r_x-r_y-s+t$ when the label noise of $x$ and $y$ is $s$ and $t$, respectively. Following Eq.(11), the ordering relationship is $x \prec y$ when $r_x-r_y-s+t < -\tau$. Since we model label noise as stochastic variables, we can compute the probability of $x \prec y$ as Formula (12) by using Eq.(2). We have clarified this point in L235-239 of the revised paper.

More comparisons

As suggested, we compare our algorithms with more noise-robust methods [1,3,4] on the MORPH II, CLAP2015, AADB, and RSNA datasets. The table below shows the comparison results on the MORPH II dataset. The proposed SOL still achieves the best scores. The results for other three datasets are similar as well. We have included all these results in the revised paper. Please see Table 1 on page 6 and Tables 12, 13, 14 on page 16 and 17.

For method RSVOR[2], we were not able to re-implement it successfully — RSVOR yields very poor results only — because its source code is not available and the authors haven’t responded to our question about implementation details. Moreover, RSVOR is designed for tabular datasets which are very different from vision datasets used in SOL. Hence, the application of RSVOR is not straightforward. We will try to fine-tune the implementation of RSVOR as much as possible and update the results.

[1] Active negative loss functions for learning with noisy labels. In NIPS, 2023.

[2] Distributed robust support vector ordinal regression under label noise. Neurocomputing, pp. 128057, 2024.

[3] Scalable penalized regression for noise detection in learning with noisy labels. In CVPR, 2022.

[4] C-Mixup: Improving generalization in regression. In NIPS, 2022.

Partially corrupted data

We measured the performances of SOL and GOL when only a small amount of samples are mislabeled. In this test, we use the CLAP2015 dataset and randomly select 10% of samples for label corruption. As shown in the table below, SOL is still effective even when a small amount of samples have label errors. We have added these results in the revised paper. Please see Table 15 and its description on page 17.

Noise distribution

We agree with you. Please note that SOL is the first attempt to extend the concept of order learning to noise-robust rank estimation, which is a relatively under-researched area. Hence, we focus on developing a simple but reasonable algorithm instead of covering all details in real-world scenarios. It is a future research issue to set the noise distribution adaptively according to each instance. We have clarified this point in the revised paper. Please see the last paragraph on page 17.

Hyper-parameters

We observed that as long as the hyper-parameters do not deviate much from the proposed default settings, decent performances are yielded. In Table 10 of page 14, we listed all the hyper-parameter values used for all datasets. Please note that the values do not vary much for the different datasets. We do agree, however, that designing an adaptive mechanism may be helpful for improving the scalability of SOL to various datasets. We will investigate it in future work. Thank you for your constructive comment.

Relabeling mechanism

During the development of SOL, we have tried a relabeling scheme where outliers are relabeled by different magnitudes. The table below compares the performances according to the relabeling scheme on CLAP2015 at $\kappa=0.4$. Although relabeling with different magnitudes also show performance improvement, the proposed technique yields better results. It is because the proposed relabeling scheme prevents drastic rank changes, as stated in L314. Please see Table 16 on page 17 for more details.

---

We have made every attempt to address your comments in the revised paper. We hope that you find this revision satisfactory. If you have additional concerns, please let us know. Thank you again for your positive comments.