ConR: Contrastive Regularizer for Deep Imbalanced Regression

Thank you very much for your thorough review and insightful feedbacks. Here we address your valuable comments:

The outermost summation in Eq. 1

Thank you very much for highlighting this point. In the design of ConR the summation over j’s is divided by the number of positive samples so that for each anchor, the average of contribution of the positive pair is incorporated into ConR. It is mistakenly dropped from Eq. 1, and has been fixed in the updated manuscript.

Input augmentations for regression loss and ConR

We appreciate your careful examination of the objective functions. Data augmentation is an indispensable component of contrastive learning and also deep visual models because it boosts robustness and generalization by bringing representational invariance to the learned models [1]. Moreover, an essential feature of ConR's design is its inherent ease of integration and orthogonality with the targeted regression baseline for regularization. Whether the regression baseline augments inputs before feeding them to L_R, our approach mirrors this process. Consequently, the incorporation of ConR requires minimal adjustments and can be seamlessly integrated without necessitating substantial considerations. Concerning L_R in our evaluation process, in all baselines and for all benchmarks, each input sample undergoes one augmentation before being incorporated into L_R. This aligns with the common practice observed in baseline methodologies. This highlights the straightforward integration of ConR into existing frameworks.\

Similarity Threshold selection

The determination of the similarity threshold is task-dependent. For instance, in the context of age estimation, employing a yearly measure is intuitively relevant. In our efforts to optimize this parameter, we conducted an ablation study ("Similarity threshold analysis" in section 4.2 of the paper and section A.5 of the appendix) to explore the impact of varying the number of years considered as the distance threshold for defining the similarity threshold. Our ablation study shows that a high similarity threshold leads to bias toward majority samples and a low similarity threshold will lead to disagreements with label relationships. This systematic examination enhances the robustness and applicability of ConR across different tasks.

Performance of ConR-only

Thank you for highlighting the importance of additional report in the experiment section. In response to your comment, in the updated manuscript we have added the performance of ConR independently, referred to as "ConR-only," in the first row of tables 1-4. This allows for a clear and comprehensive understanding of ConR's effectiveness in isolation, without the influence of additional techniques.

Representations of Balanced MSE and RankSim

Thank you for your attention. Due to space constraints, we omitted the visualizations of Balanced MSE and RankSim in Figure 4. For the representations of Balanced MSE[2] and RankSim, as well as their respective analyses, please refer to Figure 13 and "Feature visualization" in the updated manuscript appendix.\ Figure 13 shows that RanKSim by imposing order relationships encourages high relative spread while Balanced MSE suffers from low relative spread. Both RankSim and Balanced MSE have high occurrences of collapses and noticeable gaps in their feature space. Comparing the representations of Balanced MSE and RankSim with the one of ConR shows that ConR learns the most effective representations.

$1/\omega$ instead of $\omega$

Figure 5 presents the ablation study on the age estimation task. To facilitate a more intuitive comparison, we employed $1/\omega$ as it represents the distance, and in the context of age estimation, the distance corresponds to age difference. The figure illustrates the optimal choice for the age difference. Moreover, to maintain consistency, we have used this notation for the ablation study on the similarity threshold for other tasks as well.

Figure 5-b: similarity threshold omega is smaller at 1

Thank you very much for raising this valid concern. It is fixed in the updated manuscript.

References

[1] Cui, Jiequan, et al. "Parametric contrastive learning.", ICCV 2021.
[2] Ren et al., "Balanced MSE for Imbalanced Visual Regression", CVPR 2022.

Thank you for your thorough review and constructive comments. Below, we provide responses to your valuable feedback:

Theoretical Insights

We appreciate your emphasis on the importance of theoretical analysis. In response to your comment, we have added Section B in the Appendix of the revised manuscript. In the added section, we derive a upper bound on the probability of mislabelling:

$\frac{1}{4N^2}\sum_{j, x_j \in A}^{2N} \sum_{q=0}^{K^-_j} \log \mathcal{S}\_{j,q} \ p(\hat Y\_j | x\_q) \le \mathcal{L}\_{ConR} + \epsilon, \quad \epsilon \overset{N\rightarrow \infty}{\rightarrow} 0$,
where $A$ is the set of selected anchors and $x_q$ is a negative sample. $p(\hat Y_j|x_q)$ is the likelihood of sample $x_q$ with an incorrect prediction $y_q \in \hat Y_j = (\hat y\_j - \omega, \hat y\_j + \omega)$. $\hat y\_j$ is the prediction of the $j^{th}$ anchor that is mistakenly similar to the prediction of its negative pairs $x\_q$. We refer to $p(\hat Y_j|x_q)$ as the probability of collapse for $x_q$. The left-hand side presents this probability for all the negative samples. The left-hand side is the probability of all collapses during the training and regarding the equation above, Convergence of $\mathcal{L}\_{ConR}$ tightens the upper bound for it. Minimizing the left-hand side can be explained with two non-disjoint scenarios: either the number of anchors or the degree of collapses is reduced. In addition, each collapse probability is weighted with $\mathcal{S}\_{j,q}$, leading to penalizing the incorrect predictions with regard to their severity. In other words, ConR penalizes the bias probabilities with a re-weighting scheme, where the weights are defined based on the agreement of the predictions and labels. This inequality illustrates that optimizing ConR will decrease the probability of mislabelling in proportion to their mislabeling degree, with a specific emphasis on minority samples. Integrating ConR into a regression task penalizes biases, resulting in a more balanced feature space.

Leveraging reweighting methods for DIR

Thank you for highlighting the significance of the complexity analysis. As it is a valid concern, we have addressed it by providing an evaluation of the time consumption of ConR in comparison to the baselines (Please refer to Section 4.1 of the paper, titled "Time Consumption Analysis”). Our analysis show that ConR is considerably more efficient compared to FDS which explicitly aligns the feature space. Moreover, time consumption of ConR is comparable to other baselines while providing considerable performance gain.

Regarding re-weighting, the empirical observations in [1] and highlighted in VIR [2], demonstrate that using empirical label distribution does not accurately reflect the real label density in regression tasks, unlike classification tasks. Thus, traditional re-weighting techniques in regression tasks face limitations. Consequently, LDS [1] and the concurrent work, VIR propose to estimate effective label distribution. Despite their valuable success, two main issues arise, specifically for complicated label spaces. These approaches rely on binning the label space to share local statistics. However, while effective in capturing local label relationships, they disregard global correspondences. Furthermore, in complex and high-dimensional label spaces, achieving effective binning and statistics sharing demands in-depth domain knowledge. Even with this knowledge, the task becomes intricate and may not easily extend to different label spaces. For instance, in age estimation, where the label space is one-dimensional, methods like LDS and VIR can manage the task. However, when examining the code-base of [1] for depth estimation on the NYUD2-DIR benchmark, we observe that they meticulously define buckets and carefully consider various factors to make LDS feasible in that particular context. Additionally, VIR is evaluated on datasets with one-dimensional label space, and the challenges and effectiveness of their method for high-dimensional label spaces are not discussed in their study. In contrast, ConR smoothly extends to high-dimensional label spaces.

Comprehensive description of datasets

Thank you for emphasizing the importance of a detailed explanation of the datasets. For comprehensive details on the AgeDB-DIR, IMDB-WIKI-DIR, NYUD2-DIR, and the creation process of MPIIGaze-DIR, we have added information to section A.2, and dataset creation details in the Appendix of the updated manuscript.

References

[1] Yang, Yuzhe, et al. "Delving into deep imbalanced regression." ICML 2021.
[2] Wang et al., Variational Imbalanced Regression: Fair Uncertainty Quantification via Probabilistic Smoothing. NeurIPS 2023.

Thanks for the authors' comprehensive responses and updates to the manuscript. I would like to suggest the following:

1. The theoretical insights presented at the end of the appendix are surprising and significant. I recommend incorporating the main theory or conclusion from this insight into the main body of the paper. There is a concern regarding the integration of this material and its impact on the paper's structure and presentation.

2. The authors offer a detailed comparison between ConR and both [DIR and VIR]. This should be included in the related works section of the main paper, as it provides a clear distinction between these methodologies.

Thank you to the authors for their thorough responses and revisions to the manuscript. In light of these changes, I have increased my scores for Soundness, Presentation, and Contribution as my concerns have been satisfactorily addressed.

However, I would advise the authors to:

1. Ensure that concerns raised by other reviewers are also fully addressed. It is important to engage with and resolve these points to strengthen the paper further.

2. Regarding the theoretical insights in the appendix, I found no errors or doubts upon review. Nonetheless, I strongly encourage the authors to re-verify the accuracy and completeness of these sections to ensure the robustness of the paper.

Your comprehensive review and insightful comments are greatly appreciated. Here we address your comments as follows:

Combining ConR with the ordinary regression loss

Thank you very much for your attention to the details of the objective function. We acknowledge that we should provide more information. First, Eq. 4 emphasis that ConR is orthogonal to deep regression methods and can be used in any regression framework with any regression objective function. Secondly, to clarify the objective, let's delve into the optimization process in the DIR task. When modeling inter-label relationships in the feature space, two crucial factors come into play: correct similarities and correct directions [1]. Taking the age estimation task as an example, consider a scenario with a 20-year-old, a 15-year-old, and a 25-year-old person. Both the 15-year-old and 25-year-old individuals have the same label similarity with the 20-year-old (i.e., a 5-year age difference); however, one is younger, and one is older. ConR imposes proper similarities, with a focus on minority samples, while ordinary regression boosts the direction signals and result in the best performance for ConR. Furthermore, RankSim [2] is also a regularizer, assuming order relationships and optimizing their proposed loss function alongside the regression loss function.

Deviation measure in the experimental results

We appreciate your valid concern. To maintain clarity in the dense tables of DIR evaluations, we opted not to include deviations directly, aiming to prevent the tables from becoming overwhelming. Instead, we have visualized the standard deviations using error bars in Figure 3 and Figure 11. We can add a detailed table of standard deviations to the supplementary material if desired.

Performance of ConR-only as a baseline

Thank you very much for highlighting this. We value your concern and added ConR-only to the tables.

Missing reference

Thank you for highlighting the missing reference. This is fixed in the updated manuscript.

Main reason for anchor selection

We appreciate your inquiry regarding the rationale behind the proposed anchor selection. Yes, ConR does not select the input sample without any negative pair as an anchor. But the selected anchors have both negative and positive pairs dissimilar to BYOL and SimSiam. The motivation of this design is multifaceted. First, as discussed in the manuscript, for categorical contrastive learning, the anchors are predefined, and it serves as a prior knowledge in addressing imbalanced distribution. However, in contrastive learning for regression, defining anchors is not feasible, especially for complicated and high-dimensional label spaces. Secondly, treating each training sample as an anchor overlooks the imbalanced distribution, resulting in a bias towards majority labels. Empirical observations, as detailed in [3], highlight a phenomenon in deep imbalanced regression where minority samples tend to collapse towards their neighboring majority samples. This occurs even when their labels are dissimilar; for instance, in the age estimation task, features representing the age range of 0 to 6 years may exhibit similarities with those of age 30. This situation leads to confusions within certain areas of the feature space. Thus, we propose to use the correspondences between label similarities and prediction similarities to identify the confusion regions during the training. These areas of confusion are selected as anchors and push the samples that mistakenly have similar features to these anchors. The reason is that due to the skewed distribution, if we select all samples as anchors and repel all negative pairs, the learning will be biased toward majority samples. Therefore, ConR uses the anchor selection as a treatment to the biases in deep imbalance regression task without any prior knowledge about the anchors or any explicit assumption about the inter-label relationships. In summary, for each training sample $x_i$ with label $y_i$ and prediction $\hat y_i$, other training samples with a label different to $y_i$ and a prediction close to $\hat y_i$ are considered as collapses due to imbalanced distribution and serve as negative samples for $x_i$. If $x_i$ has no negative pair, it means there are no confusion around $(x_i,y_i)$ to be addressed; thus $x_i$ is not considered as an anchor and does not contribute to ConR. Otherwise, $x_i$ is an anchor and ConR pushes away samples collapsed to $x_i$ relative to label similarities to address these confusions.

References

[1] Li, Wanhua, et al. "Bridgenet: A continuity-aware probabilistic network for age estimation." CVPR 2019.
[2] Gong, et al. "RankSim: Ranking similarity regularization for deep imbalanced regression." ICML 2022.
[3] Yang, Yuzhe, et al. "Delving into deep imbalanced regression." ICML 2021.

Thank you sincerely for your comprehensive review and invaluable insights. In the following we provide the response to your comments:

Effectiveness of ConR for Deep Imbalanced Regression

We appreciate your attention to the contribution of ConR to the imbalanced learning. Here we provide clarifications as follows: Contrastive learning techniques have shown promising potentials in learning semantically rich representations by effectively disentangling essential variations from irrelevant ones. Emphasizing contrastive optimization in favor of minority samples to encourages balanced feature space is a common practice in imbalanced classification[1]. These approaches employ predefined anchors and statistical priors that are not extendible to the regression tasks. By incorporation continuity into the contrastive learning, the variations in the feature space will be modeled according to the continuity in the label space. Consequently, each sample is accurately mapped to the feature manifold in agreement with its label region on the label manifold, making it easier for the model to traverse the regions where data might be scarce due to imbalanced distribution. In addition, ConR emphasizes on modeling continuous relationships for minority samples via dynamically selecting the region of confusion (The anchor selection and negative pair selection in ConR) in the feature space to enforce variations in accordance with the label similarities (The relative pushing in ConR). Moreover, in section B of appendix (Added to the updated manuscript), as theoretical analysis shows, by optimizing ConR, the probability of incorrect labeling specifically for minority samples is reduced and ConR helps regularizing the regression process.

Comparison with Rank-N-Contrast

Thank you for your valuable suggestion regarding the comparison with Rank-N-Contrast [2]. We added Rank-N-Contrast as a concurrent work in our updated manuscript (Please refer to our latest manuscript). Their code became available in October, which is after our ICLR 2024 submission deadline, making a direct comparison infeasible. We acknowledge the importance of comprehensive comparisons; however, providing comparative results takes longer than the short timeframe of the Discussion period. Nonetheless, we plan to include 'Rank-N-Contrast' as a concurrent baseline and provide the necessary comparison results. Additionally, 'Rank-N-Contrast' is designed based on the order assumption in the label space, similar to RankSim[3]. As mentioned in our paper, this assumption is not universally valid for all label spaces. For instance in depth estimation task, 'Rank-N-Contrast' and RankSim cannot be used. However our proposed method, ConR, does not make any assumptions about inter-label relationships.

Theoretical Insights

We appreciate your emphasis on the importance of theoretical analysis. In response to your comment, we have added Section B in the Appendix of the revised manuscript. In the added section, we derive a upper bound on the probability of mislabelling:

$\frac{1}{4N^2}\sum_{j, x_j \in A}^{2N} \sum_{q=0}^{K^-_j} \log \mathcal{S}\_{j,q} \ p(\hat Y\_j | x\_q) \le \mathcal{L}\_{ConR} + \epsilon, \quad \epsilon \overset{N\rightarrow \infty}{\rightarrow} 0$,
where $A$ is the set of selected anchors and $x_q$ is a negative sample. $p(\hat Y_j|x_q)$ is the likelihood of sample $x_q$ with an incorrect prediction $y_q \in \hat Y_j = (\hat y\_j - \omega, \hat y\_j + \omega)$. $\hat y\_j$ is the prediction of the $j^{th}$ anchor that is mistakenly similar to the prediction of its negative pairs $x\_q$. We refer to $p(\hat Y_j|x_q)$ as the probability of collapse for $x_q$. The left-hand side presents this probability for all the negative samples. The left-hand side is the probability of all collapses during the training and regarding the equation above, Convergence of $\mathcal{L}\_{ConR}$ tightens the upper bound for it. Minimizing the left-hand side can be explained with two non-disjoint scenarios: either the number of anchors or the degree of collapses is reduced. In addition, each collapse probability is weighted with $\mathcal{S}\_{j,q}$, leading to penalizing the incorrect predictions with regard to their severity. In other words, ConR penalizes the bias probabilities with a re-weighting scheme, where the weights are defined based on the agreement of the predictions and labels. This inequality illustrates that optimizing ConR will decrease the probability of mislabelling in proportion to their mislabeling degree, with a specific emphasis on minority samples. Integrating ConR into a regression task penalizes biases, resulting in a more balanced feature space.

References

[1] Cui, Jiequan, et al. "Parametric contrastive learning.", ICCV 2021.
[2] Zha, Kaiwen, et al. "Rank-N-Contrast: Learning Continuous Representations for Regression." Neurips 2023.
[3] Gong, et al. "RankSim: Ranking similarity regularization for deep imbalanced regression." ICML 2022.

Three reviewers rated "marginally above the threshold" and one rated "accept". I won't be championing the paper.