Gradient Aligned Regression via Pairwise Losses

Thank you for your invaluable comments! Here are our responses:

1. "The proposed method mainly focuses on the clean data setting, while other baselines in the experiments, for example, RNC, have applications in improving generalization and robustness."

   Response: RNC is a nice regression method that incorporates pairwise label similarities with contrastive learning style method in latent feature space. It enjoys better generalization and robustness with $\delta$-ordered feature embedding, which regularizes hypothesis space to be smaller. Although we limit our scope on clean data setting, we want to highlight the following points:

   i) The proposed GAR is a novel loss function for regression and RNC is a kind of regularization on latent feature space. Those methods regularizing the latent feature space, such as RNC, are complementary to GAR, as our results on AgeDB (Table 1) demonstrate that GAR with RNC can enjoy better performance.

   ii) Different with the latent feature space based methods, GAR is proposed with more focus on the computational efficiency and theoretical insights (Theorem 1, Corollary 3 and Theorem 4), which demonstrates the unique advantages for GAR among the other pairwise regression methods.

2. "$\mathcal L_{\text{diffnorm}}$ is claimed to be a relaxed version of $\mathcal L_{\text{diff}}$. It is unclear why the objective function includes both $\mathcal L_{\text{diffnorm}}$ and $\mathcal L_{\text{diff}}$, rather than just $\mathcal L_{\text{diffnorm}}$"

   Response: As the Theorem 4 shows that fitting the label difference equals to fitting the gradients of the underlying ground truth function, therefore $\mathcal L_{\text{diff}}$ is optimized to fit $\nabla\mathcal Y$ and the relaxed version $\mathcal L_{\text{diffnorm}}$ is optimized to fit the normalized $\nabla\mathcal Y$ (captures the function shape).

   If we only include $\mathcal L_{\text{diffnorm}}$ to fit the normalized $\nabla\mathcal Y$, the objective function only could capture the function shape of $\mathcal Y$ and could lose the mechanism on capturing the original $\nabla\mathcal Y$ with magnitude information on gradients.

   On the other hand, adding $\mathcal L_{\text{diffnorm}}$ doesn't involve any extra hyper-parameter tuning burden, as we balance all the terms by a DRO framework (Theorem 5). Besides, our ablation study (Table 5 in the appendix) shows that including both $\mathcal L_{\text{diff}}$ and $\mathcal L_{\text{diffnorm}}$ achieves the best overall performance. Only including $\mathcal L_{\text{diffnorm}}$ is the suboptimal (although it is the second best).

3. "Other Comments Or Suggestions: 1. $\mathcal L_{\text{diff}}^{\text{MSE}}$ in Theorem 1 is not defined. 2. $L_{\text{diff}}$ should be $\mathcal L_{\text{diff}}$ at Line 111."

   Response: We appreciate your careful checks on our writing to improve the readability.

   We will update $\mathcal L_{\text{diff}}^{\text{MSE}}\coloneqq\frac{1}{N^2}\sum_{i=1}^{N}\sum_{j=1}^N \ell_{\frac{1}{2}\text{MSE}}(f(\mathbf x_i)-f(\mathbf x_j), \mathcal Y(\mathbf x_i)-\mathcal Y(\mathbf x_j))$ right above Eq 4. We will correct the typo at Line 111 as $\mathcal L_{\text{diff}}$.

Thank you again for providing thoughtful comments! Please let us know if our responses address your concerns.

We sincerely thank you for your invaluable feedback on our manuscript! Here are our responses:

1. "However, the significance of how the proposed method is located in the literature is not very clear for readers who's not familiar with the specific literature of using pairwise data for regression. For example, being said "However, these methods heuristically approximate label similarity information as a relative rank given an anchor data point, which may lead to the loss of some original pairwise quantitative information.", it would be desirable to have more intuitive demonstration on the weakness of existing methods. What does the anchor data point mean in this case?"

   Response: Thank you for pointing out this for us to improve the readability for the broader audience. We plan to modify this part on introduction as follows:

   For those latent space based pairwise regression methods, they define an 'anchor data point', which either act as a reference point for calculating the relative ranks to the anchor for the other data (Gong et al., 2022), or the ranks are consequently utilized to construct positive and negative pairs (Zha et al.,2023; Keramati et al., 2023). We refer the readers to the related work section (line 62, right column) for more information about those methods.

   One of the imperfectnesses for those approaches is conceptually to see: the original pairwise label similarities (like the pairwise label differences in our work) are converted to ranks or positive/negative pairs. There is an approximation loss because the conversion is single directional (and the original label similarities cannot be recovered from the coarser converted information).

Please let us know if our explanation addresses your question. Thank you again for your useful comments!

We appreciate your invaluable comments for our manuscript! Here are our responses:

1. "The cited works primarily focus on regression. However, similar types of loss functions have also been explored in the context of AUC optimization..."

   Response: we sincerely thank you for raising this point and we agree that the formulations share a similarity between the proposed pairwise regression losses and the pairwise losses utilized for AUROC optimization. Nevertheless, we will also highlight the difference: the propose pairwise regression losses do not define positive/negative pairs and the target is continuous. We will add the discussion in the related work section with the recommended reference.

2. "In this paper, the target is assumed to be a deterministic function $\mathcal Y$. This assumption is more restrictive than the commonly considered stochastic function setting. Generalizing the results to the stochastic case appears to be non-trivial, as key conclusions (e.g., Theorem 4) rely on this deterministic assumption."

   Response: Thank you for pointing out the implicit assumption and the potential extension on our current Theorem 4. Yes, its current proof relies on the assumption that $\mathcal Y(\mathbf x)$ is deterministic. We will explicitly add the assumption in our updated manuscript. It is non-trivial to extend to arbitrarily general stochastic $\mathcal Y(\mathbf x)$ setting but we're happy to provide the following extensive discussion and an extended theorem that works on the stochastic function under certain forms (should cover the "commonly considered stochastic function setting" as you mentioned) by expectation trick.

   If $\mathcal Y(\mathbf x)$ is stochastic, the $\nabla\mathcal Y(\mathbf x)$ would also be stochastic. Therefore, the deterministic model $\nabla f(\mathbf x)$ inherently could not capture stochastic $\nabla\mathcal Y(\mathbf x)$. A simple example could be $\mathcal Y(\mathbf x, z) = m(\mathbf x) +z\cdot s(\mathbf x)$, where $z\sim N(0, 1)$ is stochastically sampled from the standard normal distribution; $m(\mathbf x)$ and $s(\mathbf x)$ represent the ground truth mean and standard deviation for general heteroskedastic case. In this case, the fitting model have to learn with the randomness (like the reparameterization trick in VAE) in order to capture the stochastic $\nabla\mathcal Y(\mathbf x, z)$. Similarly as Theorem 4, we can try to define the extended theorem: $f(\mathbf x_1, z_1) - f(\mathbf x_2, z_2)=\mathcal Y(\mathbf x_1, z_1) - \mathcal Y(\mathbf x_2, z_2), \forall (\mathbf x_1, \mathcal Y(\mathbf x_1, z_1)), (\mathbf x_2, \mathcal Y(\mathbf x_2, z_2))\in\mathcal D; \forall z_1,z_2\sim N(0,1),$ iif $\nabla^kf(\mathbf x, z)=\nabla^k\mathcal Y(\mathbf x, z),\forall (\mathbf x, \mathcal Y(\mathbf x, z))\in\mathcal D; \forall z\sim N(0,1)$.

   We still could use the same logic for the proof by treating $z$ as an extra variable. However, the difficulty is learning the stochastic $z\cdot s(\mathbf x)$ term for the model because $z$ is unknown during training (or even assume the distribution for $z$ is known, the sampled value is still unknown). $f(\mathbf x, z)$ may not converge to $\mathcal Y(\mathbf x, z)$.

   However, we could be more conservative and only argue the model can capture the expected gradient on $\mathbf x$, i.e. $\mathbb E_z\big[\nabla_{\mathbf x}\mathcal Y(\mathbf x, z)\big]$, where deterministic model $f(\mathbf x)$ is competent. We can try to define the extended theorem: $f(\mathbf x_1) - f(\mathbf x_2)=\mathbb E_{z_1}\big[\mathcal Y(\mathbf x_1, z_1)\big]- \mathbb E_{z_2}\big[\mathcal Y(\mathbf x_2, z_2)\big], \forall (\mathbf x_1, \mathcal Y(\mathbf x_1, z_1)), (\mathbf x_2, \mathcal Y(\mathbf x_2, z_2))\in\mathcal D; \forall z_1,z_2\sim N(0,1),$ iif $\nabla^kf(\mathbf x)=\mathbb E_z\big[\nabla^k_{\mathbf x}\mathcal Y(\mathbf x, z)\big],\forall (\mathbf x, \mathcal Y(\mathbf x, z))\in\mathcal D; \forall z\sim N(0,1)$.

   Notice that $\mathbb E_{z}\big[\mathcal Y(\mathbf x, z)\big]=m(\mathbf x)$, $\mathbb E_z\big[\nabla^k_{\mathbf x}\mathcal Y(\mathbf x, z)\big]=\nabla^km(\mathbf x)$. Our previous proof still goes through by replace $\mathcal Y(\mathbf x)$ as $m(\mathbf x)$. Actually, this extended theorem works for any stochastic function as far as the stochastic term can be decoupled as 0 mean and can be canceled under expectation.

   We will add this discussion and extended theorem to the updated manuscript!

Thanks again for your constructive comments! Please let us know if our responses address your questions.