Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Model

We appreciate your careful reading of our manuscript and your insightful comments.

[W1] Theoretical limitation regarding learnable thresholds

[A1] We recognize this as a limitation of our results. In the case of learnable thresholds, the outcome likely depends on how the thresholds are optimized. We conjecture that if the learning of thresholds proceeds sufficiently slowly compared to the learning of $\boldsymbol{w}$ and $\boldsymbol{h}$, ONC3 may still hold. However, this setup deviates from standard DNN training practices, and we have not conducted experiments under such conditions. As this consideration clarifies, the emergence of ONC3 would depend on the details of the learning dynamics, and hence it would be nontrivial to resolve this issue. Accordingly, we would like to leave this point as a direction for future work.

[W2, Q1] Clarification on the purpose and interpretation of Figure 3

[A2] We apologize that the explanation of Figure 3 was insufficient and confusing. The purpose of Figure 3 is to intuitively visualize the occurrence of ONC1, ONC2, and ONC3. Figure 3 is divided into upper and lower parts, corresponding to the fixed and learnable thresholds, respectively. Each threshold setting is further divided into upper and lower panels, with the upper ones representing the latent space and the lower ones expressing the feature space.

Feature Space: This refers to the space where $\boldsymbol{h}$ resides (via PCA dimensionality reduction). Different colors represent different classes, and pentagrams represent the class means. Black arrows represent the classifier weights $\boldsymbol{w}$ in the formula. At epoch 0, features are scattered in the feature space; as training progresses, features collapse to their class means, reflecting ONC1. The class means then collapse to a one-dimensional subspace aligned with $\boldsymbol{w}$, reflecting ONC2.

Latent Space: This refers to the one-dimensional space where $\boldsymbol{z}$ and $\boldsymbol{b}$ in the formula reside. Different colored points represent $z$ of different classes. Thresholds $\boldsymbol{b}$ are represented by red dashed lines. At epoch 5000, the $z$ class means collapse in class order and are positioned between the appropriate thresholds, which directly reflects the occurrence of ONC3. For the fixed thresholds case, the thresholds are (approximately) located at the midpoints of adjacent $z$ class means. For the learnable case, since the boundary thresholds are set to negative and positive infinity, the $z$ class means at both ends do not show an apparent collapse. This explains why the ONC_3 metric curves converge to different values for the two threshold strategies.

We will improve the main text to clarify the above points in the revision.

[W3] Insufficient Dataset Diversity in Main Text

[A3] Thank you for this excellent suggestion. In response to [W4, Q2], we have added experiments on UTKFace, a classic facial age estimation dataset in ordinal regression tasks, with three additional backbones (details in [A4]). If the paper is accepted, we plan to include partial results from the UTKFace experiments in the main text, which would be more convincing than simply moving results of one dataset from the current appendix to the main text.

[W4, Q2] Evaluation of robustness of the theoretical findings under more challenging and realistic conditions

[A4] Thank you for your constructive suggestion. Reviewers zTjz and 7zzE also emphasized the importance of validating the ONC phenomenon on more challenging tasks. Therefore, we have examined three different backbone networks (ResNet-50, ResNet-101, and DenseNet-201) on the UTKFace age-estimation dataset binned into 20 ordinal classes (5-year intervals, 0–4 yrs up to 95–116 yrs): the maximum class-imbalance ratio is approximately 75:1. Facial age estimation using the UTKFace dataset is one of the most renowned tasks in the ordinal regression field, and the ONC phenomenon in this task will provide more robust validation for our theoretical findings.

The experimental results are summarized in Tables 1 and 2 in our response to Reviewer 7zzE (due to character limitations in this response, it was not possible to show it here). All four ONC metrics show substantial decreases across architectures, indicating successful collapse during training for both training and validation datasets in all cases.

If the paper is accepted, all these additional experimental results will be added to the paper.

[W5, Q3] Use of non-standard ordinal evaluation metrics [Q4]

[A5] We thank you for your suggestion. Reviewer 7zzE also provided a similar suggestion. According to these comments, we have additionally computed five additional metrics, including QWK, minimum sensitivity, and accuracy within 1-off.

Using the five tasks presented in the original manuscript, we have computed these metrics and found that the mean values of QWK, minimum sensitivity, and accuracy within 1-off almost converge to 1 with variance close to 0 at the final epoch. The same observation is obtained for the supplementary experiments mentioned in [A4]. Table 1 in our response to Reviewer 7zzE reports this. Notably, the learnable threshold method consistently achieves 0 minimum sensitivity on UTKFace, while the fixed threshold method overcomes this issue, implying that the fixed threshold strategy actually mitigate the minority class problem.

In the revision, we will add plots of all these new metrics to help readers better judge the training and validation situations.

[W6, Q4] Exploration of alternative ordinal loss functions beyond standard CLM framework

[A6] Indeed, changing the loss function would naturally affect the dynamics of ONC. However, our analysis is not concerning the dynamics, but rather focuses on the stationary properties of solutions obtained by minimizing the loss function. Moreover, our framework is fundamentally grounded in a statistical perspective based on maximum likelihood estimation. While it is interesting to explore the implications of using QWK, particularly in terms of its impact on learning dynamics, we consider such investigations to be beyond the scope of the present study and therefore leave them out of consideration. We would appreciate your understanding.

[W7, Q5] Limited exploration of alternative link functions suitable for skewed class distributions

[A7] As the complementary log-log (cloglog) link is associated with a log‑concave probability density function, it satisfies our theoretical assumptions and thus falls within the scope of our theory. We will stress this point in the revision.

To validate its empirical effect, we have conducted additional experiments using the cloglog link function, with the same architecture and datasets as in the original manuscript. The result is summarized in Table 1. A noteworthy point of the cloglog function is that it is not symmetric, in contrast to the logistic and normal CDF functions, and hence the equation (21) does not hold. Instead, the optimal $z_q^*$ at the weak regularization limit should be computed by numerically solving the equation (18), and thus the appropriate metric for ONC3 (equation (25)) should be modified. The result using the modified ONC3 metric (written as onc3_cloglog) is reported in Table 1.

From the result, we can see that all the three characteristics of ONC actually occur in the cloglog function as well. We will add this result in the appendix in the revision.

Table 1: ONC Metrics Evolution for LE Dataset (Initialization → Post-training)

Thank you for your valuable and constructive feedback.

[W1] Limited Practical Insights

[A1] The purpose of this paper is to investigate the existence of NC-like phenomenon in OR tasks and to provide its characterization. Hence, providing practical utility is indeed not the primary purpose of this paper. We hope for your understanding in this regard. However, our results also provide practical insights, such as suggesting appropriate ranges for regularization parameters, indicating the potential for designing new loss functions based on UFM analysis, and highlighting the usefulness of fixed thresholds. While we have already addressed these points in the Discussion section, we will revise the text to make these contributions more explicit.

[W2] Concerns About Task Simplicity

[A2] Similar comments are also raised by Reviewers 7zzE and wf6P, and we agree with this point. To address this weakness, we have conducted additional experiments using the UTKFace age-estimation dataset with 20 ordinal classes (5-year intervals, 0–4 yrs up to 95–116 yrs), which exhibits a strong class imbalance. For treating this complicated task, we have used more powerful backbone architectures (ResNet-50, ResNet-101, and DenseNet-201). All architectures utilize ImageNet pretrained weights and are fine-tuned with the Adam optimizer (lr=0.001, weight_decay=0.001). We employ the logit link function with both fixed thresholds (uniformly spaced from -40 to 40) and learnable thresholds. Tables 1 and 2 present the summarized results. Across all architectures, the four ONC metrics demonstrate significant reductions, confirming successful ONC. This consistent emergence of collapse indicates that the ONC phenomenon represents a fundamental characteristic of over-parameterized neural networks. As a noteworthy outcome, Table 1 reveals that learnable thresholds cause the model to neglect minority classes. Conversely, fixed thresholds mitigate this problem (as evidenced by the "min sensitivity" metric). This is expected and already argued in the Discussion section of the original manuscript, and the expectation has now been experimentally confirmed, providing one reasonable response to your comment regarding the weakness of our paper ([W1]). We will report this finding in the revised manuscript.

Table 1: OR Metrics After Training

Table 2: ONC Metrics Evolution (Initialization → Post-training)

Thank you for the thoughtful suggestions and detailed critique, which helped us improve the paper.

[W1] Generalization to standard neural network architectures

[A1] We sincerely thank you for this valuable feedback. To address this concern, we have conducted additional experiments. We have examined three different backbone networks (ResNet-50, ResNet-101, and DenseNet-201) and the UTKFace age-estimation dataset binned into 20 ordinal classes (5-year intervals, 0–4 yrs up to 95–116 yrs). All models are pretrained using ImageNet and the Adam optimizer (lr=0.001, weight_decay=0.001) is used for fine-tuning. The logit link function is used with fixed thresholds (uniformly spaced from -40 to 40) and learnable thresholds. The result is summarized in Table 2. All four ONC metrics show substantial decreases across architectures, indicating successful collapse during training for both training and validation datasets. The consistent emergence of collapse patterns across different architectures suggests that the ONC phenomenon is a general characteristic of over-parameterized neural networks.

If the paper is accepted, all the additional experimental results will be added to the paper's appendix. Besides the above three additional backbone networks, we also plan to supplement experimental results of transformer-based backbones. That experiment is currently underway.

[W2] Clarification on evaluation metrics

[A2] We thank you for pointing out the incompleteness of our evaluation metrics. We have additionally computed five metrics: classification accuracy, MAE of the order of predicted and true labels, QWK, minimum sensitivity, and accuracy within 1-off (mentioned by Reviewer wf6P). We will add plots of all these new evaluation metrics to help readers better judge the training and validation situations if the paper is accepted.

[W3] Improving clarity of Figure 2

[A3] Thank you for this suggestion. In the revision, we will consistently use a log scale in all the plots in Fig. 2.

[Q1] Practical implications of our theoretical findings

[A4] This is an interesting question, and it may be possible to address it through the appropriate design of thresholds or the development of new loss functions based on the UFM framework. While we have already mentioned the issue of degraded accuracy for minority classes in the Discussion section of the original manuscript, additional experiments presented in Table 1 show that with learnable thresholds, the model indeed tends to ignore the minority classes. In contrast, this issue is alleviated under fixed thresholds (see the “min sensitivity” metric). We plan to discuss this point in greater depth in the revised version.

[Q2] Justification of experimental design

[A5] In the previous experimental design, it was indeed unclear how relevant our findings were to more realistic scenarios. In response to your suggestion, we conducted additional experiments, which we believe have significantly strengthened our claims. We appreciate your valuable feedback.

Table 1: OR Metrics After Training

Table 2: ONC Metrics Evolution (Initialization → Post-training)

We are grateful for your constructive criticism, which provides an opportunity to clarify our contribution.

[W1] Limited prescriptions for training methods

[A1] Indeed, our theoretical analysis does not directly address training methods. However, UFM-based analyses can inspire practical training methods through their insightful analytical results. In fact, several prior studies based on UFM have proposed some practical prescriptions, such as the use of a fixed ETF classifier to counter class imbalance or the introduction of new loss functions to accelerate convergence; associated experiments have demonstrated that such prescriptions are actually effective. Our paper also leads to some operational prescriptions: one such example is the use of fixed thresholds to preserve ONC3 and stabilize training, which, moreover, is likely to alleviate the common issue of degraded classification performance for minority classes in the presence of class imbalance, as argued in the Discussion section of the manuscript. When submitting this manuscript, this last benefit in the imbalance case was just an expectation, but inspired by the comments of reviewers including you, we have conducted additional experiments using a more challenging task (UTKFace age-estimation dataset which has a strong class imbalance) and found that the fixed thresholds indeed improve the minority class accuracy. This is, in our view, a practical outcome naturally deduced from our theoretical arguments.

Admittedly, UFM introduces a strong simplifying assumption at the beginning. But thanks to that simplification, the derived results are often transparent. They thus can reveal what the model is doing and where to intervene, which potentially yields concrete design principles for training DNNs. This is precisely one of the key advantages of the UFM framework and is something difficult to achieve with other approaches.

[W2] Limited label scale of experimental validation

[A2] We admit that our experiments only treated the cases with a small number of classes. To address this point, as also explained in [A1], we have conducted additional experiments using the UTKFace age-estimation dataset binned into 20 classes; three additional model architectures have also been treated in the experiments to assess the robustness of the results. For more details of the experimental setup, please see our response [A1] for Reviewer 7zzE. The result clearly shows ONC in such a case with a large number of classes, supporting the practicality of our finding.

[Q1] The necessity of weight decay in UFM

[A3] Definitely necessary. In ONC2, the feature vector components orthogonal to the classifier weight collapse to the zero vector thanks to the regularization on the feature $\boldsymbol{h}$​. The same is true for ONC1: without the regularization, it is not possible to show the strong convexity of the total loss function, and hence the uniqueness of the solution leading to ONC1 is not shown. As for the regularization on the weight vector $\boldsymbol{w}$, it is necessary since otherwise $\boldsymbol{w}$ diverges, as observed in the right panel of Figure 1. Therefore, to ensure that the problem is mathematically well-defined, those regularizations are necessary.

[Q2] OR vs standard classification comparison from NC perspective

[A4] We did not consider this perspective so far. Thank you for your insightful comment. Indeed, since OR tasks can be approached as classification problems, a comparison of the two approaches could be feasible. We have examined this point from a theoretical viewpoint, but we could not draw any conclusion regarding which approach is better in this short time. A detailed investigation and comparative experiments would be necessary, and thus we would like to leave this as a direction for future work.

[Q3] UFM analysis insights for choice of thresholds

[A5] We believe that this is feasible. For example, in the class-balanced case, it is natural to assign equal probabilities to each class. Once the inverse link function $g$ and the number of classes $Q$ are fixed, the appropriate placement of thresholds $\boldsymbol{b}$ is uniquely determined. However, in such cases, we have $b_0 = -\infty$ and $b_Q = +\infty$, which implies that in the context of ONC, the optimal latent values $z_1^{\ast}$ and $z_Q^{\ast}$ diverge in the weak regularization limit. This divergence could make deep-learning-based experiments less aligned with the theoretical predictions, and therefore, we fix those edge threshold values at large enough values in the experiments, to make the ignored tail probabilities small enough while keeping the alignment with the theory. We believe that this is one of the best practices in the class-balanced setting in that the result is theoretically well-controlled, but still, the classification performance is optimal.

In the case of class imbalance, one can also assign probability values to each class based on some criterion (e.g., setting (the probability) $\times$ (the imbalance ratio) becomes constant), and then determine the corresponding appropriate values for $\boldsymbol{b}$. The corresponding ideal latent values $\boldsymbol{z}^*$ can be computed using the UFM framework. While we already discussed this point in the current manuscript’s Discussion section, we will make an effort to clarify it further in the revised version.