RankUp: Boosting Semi-Supervised Regression with an Auxiliary Ranking Classifier

Thank you for your thoughtful review and highlighting the strengths of our approach. We appreciate your careful consideration, particularly regarding augmentation. We'd like to address your concerns and questions:

W1) Influence of augmentation on results

We confirm that weak augmentation was applied consistently to the labeled data across all semi-supervised and supervised learning algorithm configurations in our evaluation. This ensures a fair comparison between our proposed method and the baselines, ensuring that performance improvements are not due to differences in augmentation strategies. We will include these details in the revised paper for better clarity. Our codes along with implementation details will be published if the paper is accepted.

To further analyze the effect of weak augmentation, we conducted a new ablation study on the UTKFace dataset with 250 labeled samples, without weak augmentation applied to the labeled data. Below is a comparison of the results:

This table illustrates the importance of weak augmentation. The performance metrics (MAE, $R^2$, SRCC) decrease when weak augmentation is not applied (please note that weak augmentation was applied on all the baselines in the paper). Despite this drop, the order of effectiveness among the methods remains consistent, whether weak augmentation is used or not. This consistency in ranking highlights the robustness of our proposed method.

W2) Lack of specificity regarding augmentation operators

We followed the settings in the USB $1$ codebase for augmentation operators, with an adjustment to the strong augmentation for audio data. This adjustment was necessary because our task involves quality assessment, and the original strong augmentation method would have affected the quality of the data. Specifically, we used the following augmentation techniques:

1. Image:

   • Weak augmentation: Random Crop, Random Horizontal Flip
   • Strong augmentation: RandAugment $2$

2. Audio:

   • Weak augmentation: Random Sub-sample
   • Strong augmentation: Random Sub-sample, Random Mask, Random Trim, Random Padding

3. Text:

   • Weak augmentation: None
   • Strong augmentation: Back-Translation $3$

Q1) Consistent application of weak augmentation

Yes, weak augmentation on the labeled data was applied consistently across all semi-supervised and supervised learning algorithm configurations in our evaluation. More information can be found in the response to W1.

Q2) Forms of augmentation in different domains

As detailed above (W2), we used domain-specific augmentation techniques based on the USB $1$ codebase, with an adjustment for audio data.

We appreciate you bringing these important points to our attention. Addressing these issues will improve the clarity and reproducibility of our work. We will revise our manuscript to incorporate these details.

---

$1$ Wang, Yidong, et al. "Usb: A unified semi-supervised learning benchmark for classification." Advances in Neural Information Processing Systems 35 (2022): 3938-3961.

$2$ Cubuk, Ekin D., et al. "Randaugment: Practical automated data augmentation with a reduced search space." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. 2020.

$3$ Xie, Qizhe, et al. "Unsupervised data augmentation for consistency training." Advances in neural information processing systems 33 (2020): 6256-6268.

Thank you for your positive comments on the novelty of our ideas and the strength of our experimental results. We appreciate your constructive feedback and will address your concerns as follows:

W1) The method description needs to be improved

We will thoroughly revise Section 3 to enhance clarity by:

1. A more detailed step-by-step explanation of our approach
2. Clearer definitions of all variables and components
3. Illustrative examples to aid understanding
4. Revise the unclear sentences.

Q1) The quality of the figures in the article needs to be improved.

We have thoroughly improved the quality of figures by:

1. Increasing the resolution of all figures
2. Enlarging the font size in all figures
3. Changing the font color to black for better readability
4. Adding more informative captions and labels
5. Unifying the size of subfigures in Figure 3

The revised figures are included in the uploaded PDF in the global response section for your review.

Q2) Several variables are unclear. For example, ℓarc, ℓregression, etc.

We will provide clear definitions for all variables in the revised manuscript. To address the specific examples you mentioned:

1. ℓarc represents the loss of the auxiliary ranking classifier (ARC).
2. ℓregression denotes the regression loss.

Q3) The description of some processes in Section 3.3 is not very clear and needs to be improved

We will thoroughly revise Section 3.3 to improve clarity by:

1. Providing a more detailed explanation of each process
2. Using consistent terminology throughout the section
3. Adding a step-by-step algorithm or flowchart to illustrate the processes
4. Including examples to demonstrate how each process works in practice

Thank you again for your constructive feedback. We believe these revisions will significantly improve the clarity and readability of our paper, making our method and contributions more accessible to readers.

Thank you for your thoughtful review and for bringing these important points to our attention. We appreciate your feedback on the clarity and technical soundness of our manuscript. Here are our responses to your concerns and questions:

W1) Regarding concerns about technical novelty

We acknowledge that RankUp and RDA are developed based on existing methods. However, our work uniquely integrates and adapts these methods for semi-supervised regression tasks. Below, we highlight the key differences between our work and the relevant existing methods:

** $1$ Burges et al., Learning to rank using gradient descent. ICML 2005:** While RankNet uses pairwise ranking loss for ranking the data, the original study focuses on information retrieval tasks. We adapt the concept of RankNet for semi-supervised regression tasks, transforming the regression problem into a classification problem. This allows us to apply existing semi-supervised classification models to regression problems, bridging a gap in the field.

** $2$ Sohn et al., FixMatch: Simplifying semi-supervised learning with consistency and confidence, NeurIPS 2020:** FixMatch is a popular method for semi-supervised classification tasks. Our work first verified that FixMatch can also be used for semi-supervised regression tasks. Additionally, we would like to point out that we adopted FixMatch as a representative method to train RankUp's auxiliary ranking classifier. Our framework is flexible and can incorporate various semi-supervised classification methods, not just FixMatch.

** $3$ Kim et al., Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning, NeurIPS 2020:**

1. Similarity: Align the pseudo-labels distribution with the labeled data distribution.
2. Difference: DARP can’t be applied to regression tasks, and thus we designed a fundamentally different approach called RDA that can be used for regression tasks. Additionally, we introduced techniques to overcome computational bottlenecks for RDA, as detailed in Section 3.3 of our paper.

** $4$ Rank-N-Contrast: Learning Continuous Representations for Regression, NeurIPS 2023:**

1. Similarity: Uses ranking loss in regression tasks.
2. Difference: Rank-N-Contrast is designed for supervised settings and can’t directly apply to semi-supervised settings. The focus of our study is semi-supervised regression tasks. Rank-N-Contrast requires the true labels of the data, whereas in a semi-supervised setting, the majority of the data is unlabeled. Moreover, one of the motivations of our proposed method is to leverage existing semi-supervised classification techniques for regression tasks, which Rank-N-Contrast can't achieve since it's a contrastive learning approach, not a classification approach.

** $5$ Gong et al., RankSim: Ranking Similarity Regularization for Deep Imbalanced Regression, ICML 2022:**

1. Similarity: Uses ranking loss in regression tasks.
2. Difference: RankSim is similar to Rank-N-Contrast, which can't directly apply to semi-supervised settings. It requires the true labels of the data, whereas in a semi-supervised setting, the majority of the data is unlabeled. More detail can reference the above response to Rank-N-Contrast.

We will expand our related work section to provide a more comprehensive discussion of existing ranking-based regression methods.

W2) Interconnection between RankUp and RDA

Although RankUp and RDA may appear unrelated at first glance, they are suitably interconnected in our semi-supervised regression framework:

1. One of the key assumptions of RDA is reliant on the ranking of the pseudo-labels. The better the quality of the ranking of pseudo-labels, the better RDA will perform.
2. RankUp enhances the ranking of pseudo-labels by incorporating ranking information.
3. The enhanced pseudo-label quality from RankUp directly benefits RDA's effectiveness, creating a synergistic relationship.

Q1) Add more discussion and ablation studies on RankUp and RDA

We appreciate your suggestion for additional ablation studies. Here are the results comparing supervised learning, RDA only, RankUp only, and RankUp + RDA on the UTKFace dataset:

These results demonstrate that:

1. RDA alone may not improve upon the supervised baseline (MAE and $R^2$ of labeled = 50 setting).
2. RankUp significantly outperforms the baseline and RDA alone.
3. The combination of RankUp and RDA yields the best performance, showcasing their synergistic relationship.

We will include this ablation study and a detailed discussion of the results in our revised manuscript. This will provide a clearer understanding of each component's contribution and their combined effect on semi-supervised regression performance.

Thank you again for your insightful comments. We believe addressing these points will significantly strengthen our paper and provide a more comprehensive understanding of our method's contributions to semi-supervised regression.

Thank you for your positive comments and insightful questions. We're pleased you found our approach novel and well-supported. Regarding the weakness and questions:

W1) Scale of the Experiments

We acknowledge that the scale of our experiments is limited, which is indeed a common challenge in semi-supervised learning for regression tasks. Traditionally, research in this area has focused on small-scale image datasets $1$

$2$ . To address this limitation and enhance the robustness of our findings, we have evaluated our methods not only on the image datasets but also across diverse datasets including image, audio, and text modalities. These varied experiments consistently demonstrated promising results, reinforcing the efficacy of our approach.

Q1) Modifications for Adapting USB Codebase for Regression Tasks

The USB $3$ codebase is designed for semi-supervised classification tasks. To adapt it for regression tasks, we made the following modifications:

1. We replaced the softmax loss function with the mean absolute error (MAE) loss function.
2. We adjusted the output layer to produce a single continuous output instead of multiple outputs used for multi-class classification.
3. For methods like Mean Teacher and $\pi$ Model, we didn't need to change the core algorithms as they are inherently applicable to regression tasks.
4. For MixMatch, we excluded components specifically designed for classification, such as sharpening and one-hot label encoding. We retained the input mixing and consistency regularization aspects, as these are valuable for regression tasks as well.

We appreciate you highlighting the need for clarity on these modifications. We will include these details in the final paper to provide a clearer understanding of our experiments.

---

$1$ Dai, Weihang, Xiaomeng Li, and Kwang-Ting Cheng. "Semi-supervised deep regression with uncertainty consistency and variational model ensembling via bayesian neural networks." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 6. 2023.

$2$ Dai, Weihang, et al. "Semi-supervised contrastive learning for deep regression with ordinal rankings from spectral seriation." Advances in Neural Information Processing Systems 36 (2023): 57087-57098.

$3$ Wang, Yidong, et al. "Usb: A unified semi-supervised learning benchmark for classification." Advances in Neural Information Processing Systems 35 (2022): 3938-3961.