RankMatch: A Novel Approach to Semi-Supervised Label Distribution Learning Leveraging Rank Correlation between Labels

W1. Code Availability and Reproducibility

RW1: We thank the reviewer for highlighting the importance of reproducibility and community impact.

We fully agree that open-sourcing the implementation is crucial for promoting transparency and encouraging further research. We are currently in the process of cleaning and organizing the codebase of RankMatch, and we will publicly release it upon the paper’s acceptance. We believe this will greatly facilitate future work on semi-supervised label distribution learning and ensure the reproducibility of our results.

Q1. On the Use of mAP and Hamming Loss

R1: We thank the reviewer for raising this insightful point. Our work is based on Label Distribution Learning (LDL), where both the supervision signal and the prediction target are soft label distributions over classes, rather than binary vectors or ranking lists as in classical multi-label learning.

Conventional multi-label evaluation metrics such as mAP, F1-score, and Hamming Loss rely on thresholding and assume discrete binary labels, making them incompatible with the goals and outputs of LDL models. In LDL, the prediction is a probability distribution over all labels (e.g., [0.3, 0.1, 0.2, 0.4]), and evaluation focuses on how close this predicted distribution is to the ground-truth distribution using distributional metrics such as KL divergence, Chebyshev, and Canberra distances.

Following standard practice in the LDL and semi-supervised LDL (SSLDL) literature [1, 2, 3], we adopted distribution-based metrics such as:

• Chebyshev distance
• Canberra distance
• Clark distance
• Kullback–Leibler (KL) divergence

These metrics are specifically designed to measure discrepancies between two probability distributions, which aligns well with the learning objectives of LDL.

That said, to further address the reviewer’s concern, we have also reported mAP and Top-1 accuracy in additional experiments (see ** Reviewer paT1 R2**). These are computed by converting soft distributions to rankings or top-K predictions. Our method achieves state-of-the-art performance under these metrics as well, demonstrating its robustness across both soft (distributional) and hard (classification-style) evaluation criteria.

[1] Facial age estimation by learning from label distributions. IEEE TPAMI

[2] Imbalanced label distribution learning. AAAI

[3] Label distribution learning. IEEE TKDE

Q2. Clarification on the Margin $\delta$ in Equation (3)

R2: We thank the reviewer for the question regarding the interpretation and sensitivity of the margin in our ranking loss formulation.

In Eq. (5), $\delta$ is not defined as the difference between the ground-truth label distributions. Instead, it is a fixed hyperparameter that serves as a confidence margin in the pairwise ranking regularization.

Otherwise, a penalty is applied to the loss. This mechanism ensures that the model not only respects the label preference implied by the distribution but does so with sufficient confidence.

We will revise the paper to clarify this definition and remove any ambiguity in notation.

Q3. Clarification on Ranking Consistency Filtering (Eq. 5)

R3: We thank the reviewer for the question. Our method filters pseudo-labels based on the consistency between predicted scores and the ground-truth soft label distribution.

Specifically, for each label pair (j, k), if the ground-truth distribution satisfies $d _{x _i}^{y^j} > d _{x _i}^{y^k}$, we expect the model prediction to follow $h _j(x _i) > h _k(x _i)$. A violation occurs when this preference is not respected.

To ensure that we only enforce meaningful and robust preference relations, we introduce a margin threshold $t$ in Eq. (5). We ignore label pairs where the difference in ground-truth importance is too small (i.e., $|d _{x _i}^{y^j} - d _{x _i}^{y^k}| \leq t$). This is important because small differences can be noisy or ambiguous, and forcing the model to learn from them may reduce robustness.

This margin helps the model focus on more confident and informative ranking signals.

W1. Structure of the Method Section

RW1: We thank the reviewer for the helpful suggestion regarding the structure of the Method section. Our intention in using multiple smaller subsections was to improve modularity and facilitate step-by-step understanding of each component in the proposed framework. However, we understand that too many headings may fragment the narrative and affect overall readability. In the final version, we will revise the section to adopt a more concise structure by merging closely related components and reducing the number of subsections, while still retaining the clarity of technical contributions. We appreciate the reviewer’s feedback and will ensure the revised version reads more fluently.

Q1. Where is the Related Work section located?

R1. We sincerely thank the reviewer for raising this important question regarding the placement of the Related Work section.

Currently, due to the strict page limit imposed by the NeurIPS submission format, we have opted to place the detailed Related Work discussion in Appendix A. This is a common practice in NeurIPS submissions, especially when the primary focus must remain on presenting novel methodology, theoretical insights, and extensive experimental results within the main paper. We have made this tradeoff to ensure that our core contributions are communicated with sufficient depth and clarity.

That said, we fully agree that making the positioning of the related work more transparent would improve the overall readability of the paper. In the camera-ready version, we plan to:

• Either integrate the essential related work into the introduction to improve contextual flow and grounding for the reader,
• Or, where formatting permits, create a dedicated Related Work section prior to the methodology section.

Additionally, we note that many of the relevant works are already cited in context throughout the introduction, method, and experiment sections to support discussion of design choices and comparisons with prior efforts. This was done intentionally to ensure readers are continuously aware of how our work relates to the broader literature.

We appreciate the reviewer’s suggestion and will revise accordingly to provide a clearer structure in the final version.

Q2. Clarification of notations in Eqs. (3)–(5)

R2: We thank the reviewer for pointing out the potential ambiguity in our notation. To enhance clarity, we will revise the manuscript to provide brief explanations near Eqs. (3)–(5). Specifically:

• Eq. (3) defines the PRR loss, a symmetric pairwise ranking regularization term that encourages the predicted scores to respect the relative ground-truth importance between label pairs $(j, k)$. It symmetrically considers both $s(j,k) \cdot g_ \delta(j,k)$ and $s(k,j) \cdot g _\delta(k,j)$, promoting consistency.

• Eq. (4) defines the binary indicator function $s(j,k)$, which equals 1 only if label $j$ is significantly more important than label $k$ in the ground-truth soft label distribution — that is, $d^{y^j} _{x _i} - d^{y^k} _{x _i} > t$, where $t$ is a small threshold. This filters out weak or noisy label preferences, enforcing only meaningful pairwise relations.

• Eq. (5) defines the penalty function $g _\delta(j,k)$, which penalizes violations between predicted scores and true label relations. Specifically, if the predicted score gap $h _j(x _i) - h _k(x _i)$ fails to meet the margin implied by the ground-truth gap $\delta = d^{y^j} _{x _i} - d^{y^k} _{x _i}$, the penalty is activated. This encourages the model to reflect label preference orderings encoded in the soft labels.

We will include a concise version of the above explanations directly in the main paper near the corresponding equations to improve readability and support understanding for readers unfamiliar with the LDL setup. We sincerely appreciate the reviewer’s suggestion.

Q3. Real-world applications of label distribution learning (LDL)

R3: We thank the reviewer for the valuable question regarding practical applications of label distribution learning (LDL). LDL is a general framework that models soft supervision, where each instance is associated with a distribution over multiple labels instead of a single hard label. This enables it to effectively capture ambiguity, uncertainty, and nuanced patterns in real-world scenarios where hard labels are either infeasible or oversimplified.

Here we briefly highlight several representative real-world tasks where LDL has demonstrated clear benefits:

• Facial age estimation: In age estimation from facial images, individuals may appear to belong to a range of plausible ages rather than a single ground-truth number. LDL allows for learning from age distributions (e.g., Gaussian over possible ages) rather than a hard label, improving robustness and interpretability [1].

• Emotion recognition: Emotional expressions often exhibit overlapping cues (e.g., “happy” with hints of “surprise”). LDL enables learning from soft emotion distributions (e.g., 30% happy, 50% surprise, 20% neutral), which reflect human perception more accurately than one-hot labels [2].

• Medical imaging: In diagnosis tasks (e.g., X-ray or MRI interpretation), multiple plausible conditions may be present, or expert disagreement may arise. LDL provides a way to encode diagnostic uncertainty via probabilistic supervision [3].

• Multi-label learning with incomplete supervision: LDL has been shown to be effective in weakly supervised multi-label tasks by recovering soft label distributions when only partial labels are observed, which is the focus of our work [4].

These examples illustrate that LDL is especially suitable for scenarios involving label ambiguity, subjectivity, or incomplete labeling, all of which are common in practical machine learning pipelines. Our method extends LDL to semi-supervised settings, further enhancing its applicability when full supervision is costly or unavailable.

[1] Facial age estimation by learning from label distributions. IEEE TPAMI.
[2] Emotion distribution learning from facial expressions. ACM MM.
[3] Class-aware multi-level attention learning for semi-supervised breast cancer diagnosis under imbalanced label distribution. Medical & Biological Engineering & Computing.

[4] Multi-label classification: An overview. Data Warehousing and Mining: Concepts, Methodologies, Tools, and Applications.

Q4. Sensitivity to Hyperparameters

R4: Thank you for the thoughtful question. We have reported detailed sensitivity analyses of key hyperparameters (λ and t) in Figure 5 of the main paper and Table 11 in Appendix D. These results demonstrate that our method maintains robust performance under a range of hyperparameter settings.

We are deeply grateful for your careful evaluation and for increasing your score. Your constructive feedback has been invaluable to improving our work.

Q1: Lack of results on PASCAL, COCO...datasets

R1: We sincerely thank the reviewer for suggesting evaluation on more standardized and larger-scale benchmarks. Our work focuses on semi-supervised label distribution learning (SSLDL), where supervision is provided in the form of soft label distributions, as opposed to the hard labels commonly used in traditional multi-label classification or semi-supervised learning.

The datasets used in our main experiments (e.g., Emotion6, Twitter-LDL, RAF-LDL, and Flickr-LDL) are widely adopted standard benchmarks in the LDL and SSLDL literature [1,2,3], as they offer soft label annotations required for this task. Unfortunately, popular datasets such as CIFAR10/100 do not natively provide label distributions and are thus not directly applicable to our setting.

To address the reviewer’s concern, we further constructed two new multi-label distribution learning (LDL) datasets from the Pascal VOC and COCO datasets, referred to as VOC-LDL and COCO-LDL, to validate the effectiveness of our method under more realistic and large-scale settings.

Notably, both Pascal VOC and COCO are originally semantic segmentation datasets with bounding box and mask annotations. We first convert them into multi-label classification datasets by treating each image as containing multiple object categories. Then, we transform these into LDL datasets by calculating the proportion of the segmentation mask area that each category occupies within an image. These proportions are then normalized to form valid label distributions. We evaluate performance using the same protocol and report results in the below.

These results clearly demonstrate that RankMatch generalizes well even on large-scale and realistic multi-label LDL datasets, significantly outperforming baselines across both evaluation metrics. The strong performance on COCO-LDL and VOC-LDL suggests that our method is not only effective in conventional LDL settings but also robust when extended to more challenging, fine-grained, and high-dimensional multi-label distribution scenarios. This further confirms the scalability and practicality of our framework in real-world applications. We will include this discussion in a future version of the paper.

Q2 Lack evaluation results such as top-1 accuracy or mAP

R2: We sincerely thank the reviewer for the insightful question regarding evaluation metrics. Our work focuses on Label Distribution Learning (LDL) and its semi-supervised variant SSLDL, where both supervision and predictions are soft label distributions rather than discrete labels. Thus, conventional classification metrics like Top-1 accuracy, precision, recall, and mAP are not directly applicable, as they assume categorical ground truths. Instead, LDL models aim to recover continuous-valued probability distributions.

Following prior works in LDL and SSLDL [1,2,3,4], we adopt distribution-based distance metrics such as Canberra, Chebyshev, Clark, and KL divergence. These are specifically designed to measure differences between distributions and better reflect prediction quality in LDL settings.

That said, for applications where hard labels may be derived (e.g., via argmax over the predicted distribution), classification metrics like accuracy or mAP can be optionally computed. However, such a process discards the rich soft distribution information, which is the essence of the LDL paradigm. To further address the reviewer’s concern regarding classification-oriented evaluation, we conducted single-label (Top-1 Accuracy) and multi-label (mAP) assessments by converting the predicted soft label distributions into discrete label sets. Specifically, we retain the original training data format using soft label distributions, while during evaluation, we transform the predicted distributions into

•	Single-label outputs by selecting the top-1 category (argmax), and

•	Multi-label outputs by selecting the top-k categories with the highest predicted probabilities.

This conversion allows us to assess classification performance while preserving the LDL training paradigm. All baseline methods are evaluated under the same protocol for fairness. The experimental results are as follows

As shown on the Flickr-LDL and Twitter-LDL datasets, our method RankMatch consistently achieves superior performance under both Top-1 Accuracy and mAP, across all label ratios (10%, 20%). This demonstrates that RankMatch not only excels at recovering accurate soft distributions, but also yields strong discrete predictions under both single-label and multi-label interpretations, showcasing its versatility and practical effectiveness. We will include this discussion in a future version of the paper.

Q3 Robustness of imbalanced data

R3: We appreciate the reviewer’s insightful comment regarding the potential mismatch in class distributions between labeled and unlabeled data. We agree that this assumption may not always hold in real-world settings, where unlabeled data often exhibit label shift.

To address this, we conducted additional experiments following the protocol of [4], which introduces a parameter $\gamma$ to control the imbalance level between labeled and unlabeled label distributions. Specifically, $\gamma$ denotes the ratio of cumulative label descriptiveness between the most and least represented classes, thereby quantifying the severity of the label distribution shift.

We constructed unlabeled subsets with varying $\gamma$ = 1, 10, 100 to simulate balanced, moderately imbalanced, and severely imbalanced conditions, respectively. The labeled data remained fixed during training. Experimental results on four benchmark LDL datasets under 10%, 20%, and 40% label ratios. To rigorously evaluate whether performance differences under different $\gamma$ values are statistically significant, we performed one-way repeated-measures ANOVA across the three \gamma settings per dataset and label ratio. This test is suitable here as all conditions (different $\gamma$) are applied to the same model and dataset settings. The experimental results are as follows

The results indicate that performance differences across $\gamma$ values are consistently not statistically significant (all p-values > 0.05). This confirms that our method is robust to label distribution shift, maintaining stable performance even under severe imbalance conditions. We will include this discussion in a future version of the paper.

[1] Predicting Label Distribution From Tie-Allowed Multi-Label Ranking. TPAMI 2023.

[2] Label Distribution Learning by Maintaining Label Ranking Relation. TKDE 2023.

[3] Fast Label Enhancement for Label Distribution Learning. TKDE 2023.

[4] Zhao X, An Y, Xu N, et al. Imbalanced label distribution learning. AAAI 2023.

Dear Reviewer paT1,

Thank you again for your valuable comments. We have been actively working to address the issues you raised in your initial review. If you have any further questions or concerns regarding our responses, we would be happy to communicate and discuss them with you during the discussion phase.

We sincerely appreciate your time and valuable feedback.

Best regards,

The Authors

W1.provide practical guidelines or heuristics for selecting λ and t under very limited labeled data

RW1: Thank you for your valuable comment. We agree that tuning hyperparameters such as λ and t may be particularly challenging when labeled data is extremely limited. To address this issue, we conducted additional experiments under the low-label setting (10% labeled data) with varying validation set sizes (5%, 10%, 15%, and 20%) to analyze the stability and robustness of hyperparameter selection.

Table 1. Validation performance under different values of λ

Table 2. Validation performance under different values of t

The results reveal the following practical guidelines:

• For λ: We observed that moderate values (e.g., λ = 0.1 ~ 0.2) generally lead to the most stable or near-optimal performance across different datasets and validation splits. This choice effectively balances regularization strength and the risk of overfitting. In contrast, λ = 0 (i.e., no regularization) usually performs poorly, especially when the validation set is small. Although λ = 1.0 can occasionally yield good results, its performance tends to be less stable and more sensitive to validation bias.

• For t: Extreme values (t = 0 or t = 1.0) typically result in suboptimal performance. In contrast, intermediate values (t = 0.2 ~ 0.5) often produce better or second-best results, particularly on the Emotion6 and RAF-LDL datasets. As the validation size increases, the performance gap between different t values narrows, but intermediate values consistently offer stronger robustness under low-resource conditions.

These findings provide empirical guidance for practitioners to tune hyperparameters effectively in low-label scenarios, especially when validation data is limited.

   W2. Lack SSLDL baseline

 RW2: We sincerely thank the reviewer for the insightful suggestion. To address the concern regarding the lack of comparison with recent SSLDL-specific baselines (e.g., IS [21] and Neu[31]), we have conducted additional experiments on these two methods. Since the official implementations of IS [21] and Neu [23] are based on MATLAB and do not support direct processing of image data, we first extracted visual features using a ResNet-50 backbone and reduced the dimensionality to 128 via PCA to fit their input format. All other experimental settings strictly follow the original paper defaults. The following table summarizes the Canberra distance (↓ lower is better) of RankMatch, IS [21], and Neu [31] across four benchmark datasets under different label ratios.

These results clearly demonstrate that our proposed method, RankMatch, consistently achieves lower Canberra distances across all datasets and label ratios when compared to the recent SSLDL-specific baselines IS [21] and Neu [31]. The performance margins are particularly pronounced on Flickr-LDL and Twitter-LDL, where RankMatch outperforms the second-best method by a notable margin, highlighting its robustness and effectiveness under limited supervision.

These additional results and analysis will be incorporated into Section 4.4: Comparison with SSLDL Methods in the revised version of the paper, along with discussions on computational setups and practical applicability.

[21] Semi-supervised label distribution learning via projection graph embedding. Information Sciences.  

[31] Semi-supervised label distribution learning with co-regularization. Neurocomputing.  

W3. Ablation study on PRR_U loss

RW3: Thank you for your suggestion. We have added an ablation study to disentangle the effect of the two components in the PRR loss. As shown below, both the unsupervised part (PRR_U) and the supervised part (PRR_L) contribute positively. Adding PRR_U already improves the results, and further including PRR_L brings additional gains. This shows both components are useful and complementary.  

We will include these results in the revised version of Table 4.      

Q1. Relation to sharpening methods

R1: Thank you for your insightful question regarding the relation between our unsupervised consistency loss (Section 2.2.1) and sharpening-based methods such as FixMatch and MixMatch. While our approach shares the overarching principle of enforcing prediction consistency under perturbations, it fundamentally differs in both the loss formulation and suitability for label distribution learning (LDL).

1. No Entropy Minimization or Confidence Filtering: Methods like FixMatch often apply entropy minimization or confidence thresholding to encourage low-entropy (i.e., nearly one-hot) pseudo-labels. However, our approach does not include any entropy-based terms. Instead, we employ KL divergence between the predicted label distributions of the original and perturbed inputs to ensure consistency. This design avoids overconfident pseudo-labels that may harm distribution modeling.

2. Incompatibility of Sharpening with Label Distribution: Sharpening functions, by design, amplify dominant labels while suppressing others, which may be beneficial for single-label classification. However, in our setting—where the goal is to predict a full label distribution that reflects label ambiguity—sharpening artificially distorts the soft structure by disproportionately down-weighting minor but meaningful label components. This leads to biased distributions that deviate from the ground-truth, especially in cases where multiple labels have comparable semantic relevance. Therefore, sharpening is not appropriate for LDL, and our consistency loss is carefully designed to preserve distributional fidelity while enhancing robustness.

Q2. Definition of δ (L102)

  R2: Thank you for pointing this out. We agree with the reviewer that the more consistent definition should be:
$\delta = d_ {\\mathbf{x} _i}^{y _j} - d _{\\mathbf{x} _i}^{y _k}$
This change better aligns with the intended meaning of comparing the distances to two labels for the same input. We will revise the notation in the final version accordingly.  

 Q3. Hyperparameter tuning

R3: Details can be find on RW1.    

Q4. Additional baselines

R4. Details can be find on RW2.

   Q5. Complexity analysis

R5: Our ensemble pseudo-label generation relies solely on forward passes with different dropout masks (or perturbations) and does not involve backpropagation or model re-training. As a result, the computational overhead is relatively low and scales linearly with the number of ensemble passes.   To illustrate, we measured the pseudo-label generation time on three datasets of varying scales using 5 ensemble rounds on a single NVIDIA RTX 3090 GPU:  

• Emotion6 (small-scale): ~1.8 seconds
• Twitter-LDL (medium-scale): ~2.5 seconds
• Flickr-LDL (large-scale): ~3.0 seconds   These results suggest that our method incurs negligible overhead, even on large-scale datasets. Moreover, this step is only performed once per iteration during training, making it practically efficient.