We would like to thank the reviewer for the valuable and constructive comments and suggestions. We have now performed new experiments and carefully revised our manuscript to address your concerns. We believe that the quality and clarity of our manuscript has been improved.

$**a) and b) More comparisons with Mixup methods and regression learning methods.**$ We thank the reviewer for pointing out the lack of comparison with other Mixup methods and regression learning methods. We firstly would like to hightlight that our SupReMix is a novel approach for $**representation learning**$ for regression tasks that stands apart from traditional $**regression learning**$ methods like C-Mixup[1]. As recommended, below we conducted two additional experiments using the AgeDB dataset: (1) we evaluated the regression performance of SupReMix (with pre-training followed by a linear probe) in comparison to Mixup methods and regression learning methods (with train from starch) including local-mixup[2], AdaMixup[3], C-Mixup[1], ordinal regression[5] and moving window regression (MWR)[4]. SupReMix has achieved better performance than $**ALL**$ of them; (from the first two parts of the table) (2) we demonstrated that integrating SupReMix with regression learning methods (with pretraining followed by linear probe with regression learning methods) can further enhance their performance (from the bottom part of the table below).

We also performed additional performance comparisons between SupReMix with C-Mixup[1] across four different datasets (see the table below). Here we show that SupReMix can consistently outperform C-Mixup train from scratch and bring improvement for C-Mixup with the SupReMix pretrain (bottom row of the table below).

c) Gain w.r.t to the Second Best. Thank you for highlighting the importance of comparing our results with the second-best method, not just the vanilla baseline. Please note that vanilla is the second-best for some datasets (AgeDB, IMDB, HCP). For clarity, we have added underscores of the second-best performance metrics in Table 1-6.

[5] Niu, Zhenxing, et al. "Ordinal regression with multiple output cnn for age estimation." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

Dear Reviewer ggUN,

As the discussion period ends in less than one day, we kindly request to know if our responses have addressed all your concerns. If so, could you please consider increasing your score? We are more than happy to provide further clarification if time allows.

Thank you for recognizing the importance of the issue we tackle, appreciating our methodology, and acknowledging the consistent improvement we have achieved. We are very grateful for your constructive comments. Below, we carefully address each of your questions to clarify and improve our submission.

Improve the clarity of our paper: As recommended, we have carefully revised the text (as well as figures and equations) to improve clarity and readability as detailed below.

• For Section 2, the main motivation is to show that the direct application of contrastive learning for classification to regression can only learn discontinuous representations that are insensitive to label permutation (Section 2.1) and swift saturation of training with conventional positive/negative contrastive pairs (Section 2.2). ("various interesting empirical studies of regression tasks, e.g., visualizations of logit distribution and latent space. These findings are supportive of the proposed method and might be inspiring for designing better algorithms for regression tasks." from reviewer 4GQV). Section 2.1 illustrates that SupCon is deficient in ordinality-awareness as it consistently generates embeddings that exhibit discrete clustering patterns, regardless of whether the labels are genuine or permuted. In Section 2.2, we observe that SupCon also lacks in presenting sufficient hardness; notably fewer contrastive pairs contribute to training during the middle or late stages. These observations directly relate to the two key aspects we previously highlighted: ordinality-awareness and hardness. To address these issues, we introduce Mix-pos and Mix-neg with weights. This enhancement aims to equip the model with a better grasp of ordinality and to amplify the level of hardness in contrastive pairs. We have modified the text to make it more explicit.
• For equation (1) and equation (3), we have made some modifications to parse them further for better readability. We change the previous set of expressions into:

$z_{m,i}^{-} = \lambda_1 \cdot z_{m,i} + (1-\lambda_1) \cdot z', \quad \text{where} \ z' \in I\backslash I_m, \ \lambda_1 \sim \text{Beta}(\alpha, \beta), \ \text{and} \ \overline{m} = \lambda_1 \cdot m + (1-\lambda_1) \cdot m'$ and $z_{m,i}^{+} = \lambda_2 \cdot z_{m',i'} + (1-\lambda_2) \cdot z_{m'',i''}, \quad \text{where} \ i'<i<i'', \ i-i', i''-i \leq \gamma, \ \text{and} \ \lambda_2 \cdot m' + (1-\lambda_2) \cdot m'' = m$

• The theoretical results in Section 3.4 are "theoretical explanations" (reviewer 4GQV) for our SupReMix loss function. In Theorem 1, the key insight is that SupReMix amplifies the penalty for more distant negative pairs compared to closer ones by assigning specific weights to these pairs. Meanwhile, Lemma 1 and Theorem 2 collectively highlight that the SupReMix loss function not only possesses a tight bound but also ensures both the ordinality and continuity of representations as they converge towards the infimum. We have concisely summarized the main takeaway in Section 3.4 to more effectively communicate the key message of this section.

Higher Dimensional Output: Thank you for this important feedback. Firstly, even if ordinality has been shown to be important in regression tasks, it is undefined for vectors in dimensions greater than one. The topology on $\mathbb{R}^n$ does not constitute an order topology, making the application of our Mix-pos method to outputs in dimensions greater than one non-trivial. In Mix-pos, obtaining a weights vector (whose dimension matches that of the label) by solving a linear system is theoretically feasible. However, this solution is not always assured due to the potential linear independence between selected samples and the anchor. And it is not scalable when increasing the dimension of regression labels. Future research could focus on exploring methods to preserve ordinality in regression representations when dealing with higher-dimensional labels. However, our Mix-neg approach and weights design can be straightforwardly applied to higher-dimensional settings. To illustrate this, we have included additional experiments to perform regression in gaze prediction (two outputs), demonstrating that our Mix-neg and weight adjustments can lead to significant improvements. We have added one more limitation section (Section E) in our submission to discuss this point.

Below we show the performance comparison of SupReMix (w/o Mix-pos) with Vanilla and other contrastive learning counterparts (SupCon and SupCR). We carried out gaze estimation tasks with the subsampled (due to the time limitation) MPIIFaceGaze dataset to evaluate the effectiveness of SupReMix. In terms of the calculation of label distance, we use cosine distance since the dimension of the target is larger than one. (More details about the experiment are in Appendix D.7.) We show that SupReMix significantly outperforms Vanilla and other contrastive learning methods.

Truly Continuous: "Could it be applied also to problems with 'truly continuous' regression targets?" The answer is a definite YES. SupReMix could be adapted to ANY bin size within the target range. In fact, smaller bin sizes are particularly advantageous as they promote greater continuity in the representations. This is due to the increased diversity of candidates available for the mixup process. Crucially, SupReMix does not depend on having samples with identical labels. It functions effectively even in scenarios where continuous variation is extreme, such as cases where each sample has a unique label. For Mix-neg, the anchor is combined with all other samples that bear different labels. In the case of Mix-pos, samples within a specified window and possessing distinct labels from the anchor are chosen for mixup. An important feature of this approach is its ability to always find a positive weight that combines a larger and a smaller label to match the anchor's label. This is achieved by simply solving a linear equation with one variable. The design of SupReMix thus ensures both flexibility and precision in handling a wide range of label variations. The bin sizes selected for our experiments were determined by the inherent characteristics of each dataset, independent of the design of our model.

Minor Things:

• For the neuroimaging datasets (HCP and UKB), we followed the previous work [1] and [2] that fine-tuned the whole network.

• Table 9 was based on AgeDB. We have added the information in the tables. Table 10 was based on AgeDB and STS-B (indicated in the header row).

• As suggested, we have changed the sentence to "Nevertheless, its dependency on data augmentation restricts its applicability to domains where effective augmentation techniques have been established, such as time series or tabular data."

• We have changed the "input" to "inputs," thank you for pointing this out.

• We have changed our title to "Mixup Your Own Pairs: Supervised Contrastive Learning for Regression with Mixup." Thank you for your excellent suggestion.

[1] Hongming Li, Theodore D Satterthwaite, and Yong Fan. Brain age prediction based on resting-state functional connectivity patterns using convolutional neural networks. ISBI 2018, pp. 101–104. [2] Benedikt Atli J ́onsson, Gyda Bjornsdottir, TE Thorgeirsson, Lotta Mar ́ıa Ellingsen, G Bragi Walters, DF Gudbjartsson, Hreinn Stefansson, Kari Stefansson, and MO Ulfarsson. Brain age prediction using deep learning uncovers associated sequence variants. Nature communications, 2019.

Thank you for recognizing the significance of the problem we address, the ingenuity of our formulation, and the importance of our method, which notably does not depend on data augmentation and demonstrates superior performance. We truly appreciate your helpful suggestions and have provided detailed responses/revisions to address your points.

More Comparisons (C-Mixup): Thank you for your constructive comment. As recommended, we have added performance comparison with C-Mixup (training-from-scratch) on four datasets. Moreover, as SupReMix is a representation learning method, it can jointly work with C-Mixup by using SupReMix pretraining and C-Mixup during the linear probe. We demonstrate that, compared to training from scratch, utilizing our SupReMix pre-training consistently yields improvements.

Pair Selection Strategy: We appreciate your in-depth advice on the relation of SupReMix with C-Mixup in terms of "pair selection strategy". In short, our pair selection strategy is in the same direction but different from C-Mixup's. Although both of us consider the label distance, C-Mixup will only put a higher probability of mixup for pairs with smaller label distances. SupReMix, as a contrastive learning method, treats positive pairs mixup (Mix-pos) and negative pairs mixup (Mix-neg) differently. More specifically, SupReMix takes a window size (γ) of label distance for Mix-pos and leaning-toward-anchor beta distribution for sampling anchor-inclusive pairs (Mix-neg). Mix-Pos strongly depends on local linearity assumption, therefore we do not consider samples with large label distance (even with low probability) for hard positive mixtures. For performance comparison, we replaced our pair selection strategy with C-Mixup's and experimented on the AgeDB dataset. We have shown that our proposed approach still outperforms C-Mixup (see the table below).

No Ordering is Needed: Thanks for raising this question. We would like to clarify that our method does not require a specific order for selecting contrastive pairs. For hard negatives, the anchor is mixed with all other samples in the same mini-batch that have differing labels. Regarding hard positives, the selection process involves identifying samples within a predefined γ window. This is achieved by first calculating the label distance between the anchor and all other samples. We then select the samples that fall within the window. This process can be efficiently implemented through matrix subtraction and identifying the minimum values.

All the backbone networks were trained from scratch (if you refer to the contrastive learning stage itself).

Higher Dimensional Output: Thank you for this important feedback. Firstly, even if ordinality has been shown to be important in regression tasks, it is undefined for vectors in dimensions greater than one. The topology on $\mathbb{R}^n$ does not constitute an order topology, making the application of our Mix-pos method to outputs in dimensions greater than one non-trivial. In Mix-pos, obtaining a weights vector (whose dimension matches that of the label) by solving a linear system is theoretically feasible. However, this solution is not always assured due to the potential linear independence between selected samples and the anchor. And it is not scalable when increasing the dimension of regression labels. Future research could focus on exploring methods to preserve ordinality in regression representations when dealing with higher-dimensional labels. However, our Mix-neg approach and weights design can be straightforwardly applied to higher-dimensional settings. To illustrate this, we have included additional experiments to perform regression in gaze prediction (two outputs), demonstrating that our Mix-neg and weight adjustments can lead to significant improvements. We have added one more limitation section (Section E) in our submission to discuss this point.

Below we show the performance comparison of SupReMix (w/o Mix-pos) with Vanilla and other contrastive learning counterparts (SupCon and SupCR). We carried our gaze estimation tasks with the subsampled (due to the time limitation) MPIIFaceGaze dataset to evaluate the effectiveness of SupReMix. In terms of the calculation of label distance, we used cosine distance since the dimension of the target is larger than one. (More details about the experiment are in Appendix D.7.) We show that SupReMix significantly outperforms Vanilla and other contrastive learning methods.

Notations: Thank you for your advice. We have made the following modifications to enhance clarity and conformity: (1) The text in equations has been altered from italic to normal font. For instance, the word "with" in Theorem 1 now appears in standard font. (2) Vectors are now in bold, aligning with common mathematical practice, e.g., vector z used for embeddings. (3) The function notation g has been replaced with a Greek letter ρ.

Figures: Thank you for your suggestion to improve the figure of schematic overview. We have made modifications to make the proposed method more explicit: (1) In panel A, we use red and green to color code the Mix-Pos and Mix-Neg; (2) In panel B, we change the names of our hard pairs to "Mix-Pos" and "Mix-Neg", to be consistent with panel A.

Thank you for recognizing the strong motivation behind our method and its significant performance improvements. We also appreciate your acknowledgment of our comprehensive empirical analysis and theoretical explanations. We have prepared the response below following your constructive comments and suggestions. We hope that the following has addressed your concerns adequately.

Novelty (W1):

We appreciate the reviewer's assessment regarding the novelty of our proposed method. It provides us with an opportunity to clarify the contributions and innovative aspects of our work. The novelty of our work can be summarized in the following three key aspects:

(1) Contribution to an Underexplored Domain: We wish to highlight that the majority of previously proposed hard sample mining methods, including those like SupCon and Decoupled CL you mentioned, predominantly focus on classification rather than regression. The direct application of them to regression leads to sub-optimal performance due to lack of ordinality-awareness and hardness. Regression is "a domain to which a lot of our insights and beliefs from classification may not generalize" (reviewer gvot) and "an important and interesting problem that I think should be studied more" (reviewer DdrH). "The idea of applying contrastive learning to regression is interesting" (reviewer ggUN). To our knowledge, SupCR is the only method centered on regression (We have thoroughly compared with it and provided analysis in our submission even if it was an Arxiv before submission), yet it faces a significant limitation: the necessity for data augmentation, which may not be suitable for modalities like time series or tabular data. SupReMix stands out as the first supervised contrastive learning framework for regression that does NOT rely on data augmentation ("This is significant" from reviewer gvot). Furthermore, earlier strategies such as MoCHi and i-Mix were implemented in a self-supervised manner, not in supervised training. Our approach utilizes regression labels to foster ordinality and continuity, aspects that were overlooked in previous methods.

(2) Technical Novelty: The prior methods in contrastive learning and hard sample mining, including all those you've mentioned like SupCon, SupCR, Decoupled CL, MoCHi, and i-Mix, primarily focus on identifying hard negatives, leaving the concept of hard positives largely unexplored. We are grateful for your recognition of our "various interesting empirical studies of regression tasks, including visualizations of logit distribution and latent space." These studies underline the importance of incorporating hard positives in regression analysis. Our approach introduces the novel hard positive pairs in regression by combining two negative samples in such a way that their convex combination of labels matches the anchor's label. It aims to promote local linearity. SupReMix represents a groundbreaking approach in addressing the issue of hard positives in the regression context and proposing a solution leading to superior regression performance. Even in the broader scope of contrastive learning in general, hard positives remain an underexplored area. Wu et al. 2023, as cited in our related work, appears to be the only study delving into this concept, albeit in a self-supervised learning framework. On the other hand, SupReMix is the first contrastive learning framework that incorporates label distance into contrastive loss, providing a more fundamental approach to handle negative pairs, and it is supported by the concept of Distance Magnifying we proposed and theoretically analyzed.

(3) Theoretical Contribution: SupReMix is supported by two theorems and a lemma we established, offering novel insights in supervised contrastive learning for regression. We demonstrate that by assigning weights to contrastive pairs based on label distance, our approach amplifies the penalty for negative pairs that are more distant relative to nearer ones. Besides, we establish that the SupReMix loss function has a tight bound, and it guarantees the ordinality and continuity of representations as it approaches the infimum.

Comparisons with related work (W2): We appreciate the important comment. We firstly would like to highlight that our SupReMix is a novel approach for $**representation learning**$ for regression tasks that stands apart from traditional $**regression learning**$ methods like C-Mixup[4]. As recommended, we conducted two new experiments to validate the effectiveness of SupReMix: (1) we evaluated the regression performance of SupReMix (with pre-training followed by a linear probe) in comparison to methods (both training-from-scratch and pretrain + linear probe) including MoCHi[2], i-Mix[3], C-Mixup[4], Vanilla Mixup, and M-Mixup[6]. SupReMix has achieved better performance than ALL of them using AgeDB dataset (see the top two parts of the table below); (2) we demonstrated that integrating SupReMix with C-Mixup can further enhance C-Mixup's performance (bottom part of the table below).

We also performed additional performance comparisons between SupReMix with C-Mixup[4] across four different datasets (see the table below). Here we show that SupReMix can consistently outperform C-Mixup train from scratch and bring improvement for C-Mixup with the SupReMix pretrain (bottom row of the table below).

Hyper-parameters (W3): We appreciate this useful point. To clarify, SupReMix introduces ONLY ONE additional hyper-parameter: window size (denoted as $\gamma$) for Mix-pos. This parameter is chosen based on the range of regression targets. Table 10 demonstrates that $\gamma$ is arguably robust: moderate ranges of $\gamma$ consistently enhance performance. Another consideration may be related to the beta distribution for Mix-neg. In Table 11, by experimenting with different shapes of distribution on six datasets with different modalities for different tasks, beta(2,8) consistently outperforms other variants. It indicates that as long as the level of difficulty is maintained at a certain level, i.e., the beta distribution leans towards the anchor, the algorithm can perform the task robustly.

Related Work (W4): Thank you for pointing out these useful references. As recommended, we have included the following sentences to highlight these studies and clarify the research gaps. (In Introduction, second paragraph).

"... Nevertheless, they mainly focused on hard negatives with hard positives underexplored, and the hardness of contrastive pairs in contrastive learning for regression remains inadequately examined. Data mixing techniques (Zhang et al., 2018; Verma et al., 2019; Shen et al., 2022), commonly used for data augmentation, have been used to create hard samples in previous contrastive learning methods for classification (Kalantidis et al., 2020; Lee et al., 2020; Liu et al., 2023). However, these methods do not leverage the label distance to differentiate the hardness among hard negative mixtures. ..."

Thank you so much for your reply! We would like to further emphasize the novelty of our method. We believe that a clearer articulation of novelty will distinctly highlight our contributions.

We agree that the concept of hard-sample mining (e.g., MoChi[6]) and the usage of label distance for mixup (e.g., C-Mixup[7]) have been studied in previous work. We would like to underscore the following key points:

1. SupReMix's Contribution on hard-sample mining: Previous hard-sample mining strategies typically “fail to address hard positives” as they cannot be applied to generate hard positives. In contrast, SupReMix introduces a unified framework that effectively generates both hard positive and negative samples. This is accomplished through our innovative anchor-inclusive and anchor-exclusive mixup strategy, integrating both ordinality and continuity into the process. To the best of our knowledge, the most relevant work, Wu et al [5] is the only one addressing hard positives. However, it is limited to image data, and the generation of hard positives requires training a GAN model which is known to be difficult to optimize.

2. Comparison of Our Approach with C-MixUp in Terms of Label Distance Usage: Unlike C-MixUp, which selects samples based on label distance (with all samples potentially being selected), our method drastically differs by leveraging label distance differently for positives and negatives.

   • For negatives, label distances serve as weights for controlling the degree of strength of repulsion. This fundamental difference in approach allows for a more nuanced treatment of negatives, beyond mere sample selection.
   • For positives, unlike C-MixUp, which gives all potential pairs a certain probability, SupReMix uses a defining window size $\gamma$, strongly dependent on the local linearity assumption. Therefore, we do not consider samples with a large label distance (even with low probability) for positives.

3. Differentiating Our Approach with Concurrent Work (We understand the concerns raised, and while we believe it is not necessary to address this point, we are more than happy to further highlight our contribution): Concurrently, we recognize similar efforts in using label distance for regression in other works. Notably, we are the first to theoretically analyze a property of contrastive regression relevant to label distance, which we have termed Distance Magnifying (DM). By applying label distance as a weight for negatives, we enhance the penalty for more distant negative pairs compared to nearer ones. Additionally, our loss function demonstrates a tight bound, ensuring the ordinality and continuity of representations as it converges to the infimum.

We thank the reviewer for acknowledging that we have solved W2, W3, W4 well. For W2, as recommended, we have added comparisons with C-Mixup[7], RoBERTa[4] into our main text. For the rest compared methods, we will add complete comparison for all six datasets for the camera-ready version.

Finetuning

Noted that RNC[1] (“SupCR” previously) is an Arxiv paper when submitting, and we have already compared it with all six datasets following their linear probe scheme (please note that the original RNC[1] paper only provides linear-probe results). However, we thank reviewers for suggesting comparing SupReMix with RNC[1] under the setting of fine-tuning. As recommended, we here provide evaluation with fine-tuning for SupCon[2]/RNC[1]/SupReMix on AgeDB and IMDB dataset. We observe that SupReMix consistently outperforms SupCon[2] and RNC[1] under the fine-tuning evaluation.

Comparing with generative pre-training baselines

We thank the reviewer for the interest in our method. We compare our method with MAE[3] and RoBERTa[4] for with linear-probe on AgeDB and STS-B respectively. SupReMix consistently outperforms these methods.

[1] Zha, Kaiwen, et al. "Rank-N-Contrast: Learning Continuous Representations for Regression." NIPS 2023.

[2] Khosla, Prannay, et al. "Supervised contrastive learning." NIPS 2020.

[3] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." CVPR 2022.

[4] RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv, 2019.

[5] Synthetic Data Can Also Teach: Synthesizing Effective Data for Unsupervised Visual Representation Learning. AAAI, 2023.

[6] Hard Negative Mixing for Contrastive Learning. In NeurIPS, 2020.

[7] C-Mixup: Improving Generalization in Regression. In NeurIPS, 2022.