Unleashing the Potential of Classification with Semantic Similarity for Deep Imbalanced Regression

• The motivation is unclear. First, the description of deep imbalance regression is too simple and it is difficult for some readers to understand the task and its scenarios. Some visual examples are expected in the introduction. Second, the phenomenon of distorting the geometrical structure of the feature space seems to be interesting, but how and why does it affect the performance of deep imbalance regression? Finally, while the authors present two soft labeling strategies based on semantic similarity to smooth the labels, they do not directly address the label imbalance. It would be better to clarify how the semantic similarity and the proposed method specifically deal with the label imbalance. Otherwise, the proposed label smoothing based on semantic similarity modifies the original one-hot label vector to a soft label. It may introduce some noises, so how do the authors deal with those noisy supervision?
• The contribution may be incremental. The paper only proposes two soft labeling strategies for group classification from the label smoothing perspective, and employs existing label distribution smoothing [1] and group contrastive representation learning [2] tricks. So, the contribution seems to be incremental.
• Some theoretical analyses are required. The paper could be strengthened by providing some theoretical analysis and insights to support the empirical findings.
• Minor issues. (1) The definition of $c(\psi)$ is missing. (2) “accurate estimating” in line 156 should be “accurate estimation”. (3) The figures in this paper are unclear.

[1] Yuzhe Yang, Kaiwen Zha, Yingcong Chen, Hao Wang, and Dina Katabi. Delving into deep imbalanced regression. In International conference on machine learning, pp. 11842–11851. PMLR, 2021.

[2] Kaiwen Zha, Peng Cao, Jeany Son, Yuzhe Yang, and Dina Katabi. Rank-n-contrast: Learning continuous representations for regression. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.

I have several concerns in the following aspects:

1. Previous work has employed smoothed labels to calculate the mean squared error (MSE) for training regression models. The authors introduce a group classifier based on this concept; however, according to the article, the group classifier operates independently from the instance regressor that predicts $y$. The group label is not directly related to the sample label, and the authors have not provided the generation process or conducted an ablation experiment for the group label. More experiments and inferences are needed to demonstrate the necessity of the proposed group classifier.

2. Some writing needs better explanation:

• The legend in Fig. 1 requires clearer color interpretation.
• The definition of the sign $k$ in Eq. 5 is missing.
• The final training objective needs clarification.
• The presence of peaks in Fig. 3(b) and (c) when the group number is 40 needs an explanation.
• Line 247: Missing part of $p(g|)$.
• Line 324: Lack of space after the '/'.

3. Further ablation studies are recommended:

• An ablation study excluding the classification term is needed.
• An ablation study excluding contrastive learning should also be included.

1. The methodology includes pseudo-labeling, label smoothing, and contrastive learning, which are widely used techniques. While the symmetric descending soft labeling strategy is interesting, it may not sufficiently support the novelty of the entire study.
2. Soft labeling plays a crucial role in the algorithm. How does the paper ensure the generation of high-quality soft labels?
3. There are differences in the comparison algorithms across the three datasets, with certain advanced baselines missing from the IMDB-WIKI-DIR and STS-B-DIR datasets.
4. Details on some hyperparameter settings are lacking. The paper does not specify values for $\beta$ in the soft label settings and the batch size for contrastive learning. Additionally, the temperature coefficient for contrastive learning is not explicitly shown in the equation.

• The proposed approach's effectiveness and soundness are not well justified, mainly because there is no theoretical proof.
• The claim in lines 60-62 states, "Therefore, semantic similarity, another aspect of the continuity in DIR where the similarity across labels would also reflect the similarity of their features, is always overlooked by the previous works." is not entirely valid, since there are several approaches modelling continuity in the feature spaces [1,2,3].
• The extension of the proposed method to multi-dimensional label spaces has not been investigated or experimented with.
• The discussion in section 2.2 is not convincing of how semantic similarity is modelled with the proposed approach.
• The ablation study is not complete.