Spectral Contrastive Regression

4. We note that the paper 'Rank-N-Contrast: Learning Continuous Representations for Regression' was published on Arxiv after the submission of our paper and remains unpublished. Given its post-submission release and unpublished status, it was not valid for comparison in our study.

Questions

Q1: FT stands for Fine-tuning; we included this explanation in Section 4.1 of the revised paper.

Q2: $L_{std}$ demonstrates better performance over the baselines in numerous instances. Please note that the mixup technique synthesizes the data with a new label distribution. When the new label distribution differs significantly from the original, the $L_{svd}$ loss might mislead the model into aligning the label distributions by constraining the scale of the output. We hypothesize that high label and sample density can prevent this issue, a condition that holds true in the presence of large datasets. Therefore, given the large size of MPI3D, $L_{svd}$ is more suitable for this dataset compared to the other seven, as detailed in Table 3 of our updated paper.

Thank you very much for your time and feedback. We incorporated your feedback in the updated paper by adding the results of RankSim and FDS to the tables. Specifically:

1. We report the results with the Feature Distribution Method (FDS) by conducting the experiments over 7 datasets. The results are shown in Tables 1 and 2 of the updated paper. As can be seen, our method outperforms FDS in most cases.
2. Unlike our approach, the paper "RankSim: Ranking Similarity Regularization for Deep Imbalanced Regression" only considers the relation between the order of feature distances and the order of label distances, and not their proportions. Therefore, it can not guarantee the Lipschitz continuous property of the predictor $p$. In other words, RankSim may lead to very steep inclines in the feature distribution, which may affect the continuity of the feature distribution. We add the results of RankSim in the tabels as well. Since the RankSim is only designed on the dataset with discrete target labels, we skip those datasets where RankSim can not be applied. The result is shown in Tables 1 and 2. Our method outperforms RankSim in most cases. Also, since RankSim has a similar motivation with $L_{std}$, we also visualize the resulting feature distribution on DTI dataset in Figure 1 of the updated paper, and compare it to our method. As can be seen, even though Ranksim has a more discriminative pattern, there are more breakpoints in the distribution because of lacking the Lipschitz continuous property. This may be the reason that our proposed $L_{std}$ can perform better than RankSim.
3. The paper "Delving into Deep Imbalanced Regression." proposed two strategies to smooth label distribution and feature distribution. Since the Label Distribution Smoothing (LDS) is applied on discrete target labels, it cannot be directly applied to our datasets where the labels have real/continues values. More details can be found at the github link https://github.com/YyzHarry/imbalanced-regression/blob/a6fdc45d45c04e6f5c40f43925bc66e580911084/agedb-dir/datasets.py#L60

• The traditional contrastive learning in classification aims to generate a more discriminative feature distribution using contrastive pairs, namely positive pairs and negative pairs. Many contrastive losses, such as triplet loss, belong to the area of metric learning and aim at minimizing the distance between positive pairs and maximizing the distance between negative pairs. In the context of regression tasks, the $L_{std}$ loss can be considered a form of metric loss, as it introduces a new metric distance $d_r$. This adaptation of the contrastive concept to regression is due to the continuous nature of the labels.

• Generating a discriminative distribution with continuous labels is a challenge in regression tasks. As discussed in the paper, the ideal situation is to have $d_r$ constant at $d_r^\ast = |{W^\ast}^{-1}_p|$. Under this framework, we can define two contrastive pair sets:

S_l = \left[(f_i, f_j)|d_r(f_i,f_j) < d_r^\ast\right]; S_g = \left[{(f_i, f_j)|d_r(f_i,f_j) > d_r^\ast\}\right]

In this case, contrastive metric learning in regression aims to minimize the variance, ensuring that the feature distance aligns more closely with the optimal proportion. This is achieved by minimizing the distance within $S_l$ (where feature pairs are closer than the ideal distance) and maximizing the distance within $S_g$ (where feature pairs are farther than the ideal distance). The $L_{std}$ loss helps to decrease unexpected variance in the feature space, thus maintaining the proportionality essential in regression tasks.

• We thank the reviewer for the suggestion and acknowledge the need for a new title to better reflect the paper's content and to distinguish it from traditional concepts of contrastive learning. We propose an updated title of "Unifying Scale and Proportion for Feature-Level Generalization in Regression", and will replace the subtitle of Section 3.2 with: "Proportional distance for Continuous Label Spaces". These changes aim to more accurately convey the focus of the paper on optimizing feature distances in regression tasks and its contribution to metric learning with continuous labels.

• The second term $L_{std}$ is calculated for the two distributions separately, represented as $L_{std}^{real}$ and $L_{std}^{syn}$, respectively. The final $L_{std}$ loss is the sum of $L_{std}^{real}$ and $L_{std}^{syn}$. The third term $L_{svd}$ is only applicable when there is more than one distribution. According to Equation 3, $L_{svd} = |max(s_{real}) - max(s_{syn})|$, where each singular value is calculated from one distribution. Therefore, there should be at least two distributions for the $L_{svd}$ term to be applicable.

Thank you very much for your time. We incorporated your feedback into the updated paper by adding more experiments. Additionally, we address your questions below.

Q1. $L_{std}$ is calculated for both the original distribution and the augmented distribution. As requested by the reviewer, we have conducted extra experiments to study the effect of augmentation and the proposed $L_{std}$. The following table shows the results of an ablation study on $L_{std}$, with and without augmentation.

As can be seen from the table, FT+$L_{std}$(w augment) always outperforms FT(w/o augment). Compared with FT(w augment), FT+$L_{std}$(w augment) brings extra 1%-3% improvement on average.

Q3.1 ``Because from Tables 1 and 2, single svd loss term can provide significant improvement and sometimes has the best performance. But when combined with std, it does not show a significant effect"

We are uncertain about your comment in Q3.1, as $L_{std}$ demonstrates better performance over the baselines in numerous instances. Please note that the mixup technique synthesizes the data with a new label distribution. When the new label distribution differs significantly from the original, the $L_{svd}$ loss might mislead the model into aligning the label distributions by constraining the scale of the output. We hypothesize that high label and sample density can prevent this issue, a condition that holds true in the presence of large datasets. Therefore, given the large size of MPI3D, $L_{svd}$ is more suitable for this dataset compared to the other seven, as detailed in Table 3 of our paper.

Q3.2 ``In Fig.2, $\beta$ introduces little to no effect on the metrics for varying its values in the entire range. Do the authors have any speculation or analysis on this pattern? "

We agree that the impact on $L_{svd}$ appears insignificant in Fig. 2, likely due to the large scale of $L_{std}$. This might give the impression that $L_{svd}$ is unaffected by $\beta$. To clarify this, we have presented the hyperparameter analyses of $\alpha$ and $\beta$ separately in the revised version of our paper. For the analysis of $\beta$, we find that setting the hyperparameter to 1000 on the DTI dataset results in better performance for $L_{svd}$. As we discussed earlier, the MPI3D dataset might be more suited for $L_{svd}$, so we have also included a sensitivity analysis of $\beta$ for the MPI3D dataset in the Appendix of the updated paper. The impact of $L_{svd}$ is significant in some cases: the model achieves the best results when $\beta$ is set around 1. However, if $\beta$ is too large, causing $L_{svd}$ to dominate the total loss, the performance worsens

Q4. We added the t-SNE visualization in the Appendix of the updated paper.

Q5. FT stands for Fine-tuning; we included this explanation in Section 4.1 of the revised paper.

Q2. There are significant differences between Ranksim and our method. RankSim considers the relation between the order of feature distance and the order of label distance instead of the proportion. So, it is hard for RankSim to keep the predictor Lipschitz continuous. That can lead to steep slope in the feature distribution, which is shown by the discontinuity in the visualization of Figure 1 of the updated paper. The following tables show the results of RankSim compared with our method. We skip the datasets RankSim can not be applied. Our method outperforms RankSim in most cases. Also, since RankSim has a similar motivation with $L_{std}$, we also visualize the resulting feature distribution on DTI dataset in Figure 1 of the updated paper, and compare it to our method. As can be seen, even though Ranksim has a more discriminative pattern, there are more breakpoints in the distribution because of lacking the Lipschitz continuous property. This may be the reason that our proposed $L_{std}$ can perform better than RankSim.