Positive-unlabeled AUC Maximization under Covariate Shift

Thank you for your positive comments and constructive feedback. We added experimental results in the anonymous URL: https://anonymous.4open.science/r/icml25s-6C74/results.pdf

Weaknesses: (1) The paper lacks a clear explanation of the motivation for deriving the two estimators of the AUC risk on the test distribution under covariate shift.

We have described the motivation for deriving the two AUC risk estimators in Section 4.2 and the fourth paragraph of Section 1. Specifically, under the covariate shift, existing AUC maximization methods cannot minimize the test AUC risk. To deal with this, we derived two AUC risk estimators that can approximate the test AUC risk. Since these estimators are calculated with different data for classifier learning (i.e., Eqs. (17) and (19)), we can use all available data $X ^{{\rm p}} _{{\rm tr}} \cup X _{{\rm tr}} \cup X _{{\rm te}}$ for classifier learning by using both estimators (i.e., Eq. (20)). We will clarify this.

there is no analysis of their error bounds.

Thank you for the important feedback. We would like to work on analyzing the error bounds in the future.

(1) Has the issue of noise been considered? When the unlabeled data contains significant noise or redundant information, what impact would this have on the performance of the proposed AUC maximization method?

Yes, we have considered the issue of noise. This is because unlabeled data can be regarded as noisy negative data since it contains a small number of positive data (noise). As class-prior $\pi$ becomes large (the noise in unlabeled data becomes large), the performance of our method tends to decrease, as described in Table 1. Nevertheless, our method outperformed the other methods.  

(2) ... in real-world applications, such inconsistencies might also manifest as changes in class distributions (positive distribution shift) or shifts in class priors. In these inconsistent scenarios, can the proposed method still effectively perform AUC maximization?

Thank you for the insightful question. Please see the answer to the last comment of Reviewer Gocn.

(1) The design in the paper relies on the sigmoid loss function. Would other loss functions be compatible with this solution approach?

We can derive the same form of the loss function of Eqs. (17) and (19) when using symmetric loss functions as described in Lines 259--267. The symmetric functions include many popular losses, such as sigmoid, ramp, and unhinged losses.

(2) Why do the PAUC results in Table 1 perform particularly poorly, even being the worst in most cases? They are troublingly bad.

PAUC strongly depends on the assumption that the negative-conditional density does not change between the training and testing to derive the loss function. However, the negative density changed significantly in our datasets. In addition, as noted in the PAUC paper, PAUC performs poorly when $\pi_{{\rm te}}$ is small due to its reliance on extracting positive data from unlabeled test data. For these reasons, PAUC did not work well in our setting. Note that we have confirmed that PAUC performs well in the setting used in the PAUC paper.

(3) The results on the SVHN dataset are consistently anomalous.

Since each dataset has different properties, it is unsurprising that the trends of the results differ across datasets. Meanwhile, the SVHN is not the only data that shows different patterns. For example, although trPU, tePU, and CPU worked relatively well in MNIST and FashionMNIST, they performed poorly (AUC is about 0.5) in both SVHN and CIFAR10 in Table 1.

(4) Fig.2 shows the importance weight distribution on the FashionMNIST dataset when the positive class prior is 0.1. Does this result suggest that the method might overly rely on the test data during training? Furthermore, how do the distributions behave when the priors are set to 0.01/0.05?

This result does not mean our method overly relies on the test data for classifier learning. This is because the (estimated) importance weights are used for only PU data in the training distribution in Eqs. (17) and (19).

As described in the second paragraph of Section 5.1, the covariate shift was created so that the high- and low-density regions of the inputs were reversed between the training and testing. Since importance weights are defined as density-ratio $w({\bf x}) = p_{{\rm te}}({\bf x})/p_{{\rm tr}}({\bf x})$, test data would tend to have larger weights than training data. (This is also valid for the relative density-ratio in Eq. (21)). Figure 2 examined whether the estimated weights correctly reflect this characteristic. When the class-priors are $0.01$ or $0.05$, similar results are obtained in Figure A in the anonymous URL. We will clarify this.

Thank you for your insightful comments and constructive feedback. We added experimental results in the anonymous URL: https://anonymous.4open.science/r/icml25s-6C74/results.pdf

The scale of the adopted datasets is too small, e.g., only "50 positive and 3,000 unlabeled data in the training distribution and 3,000 unlabeled data in the test distribution". In fact, for benchmarks like MNIST/CIFAR10, the authors could use much more examples (e.g., 10000+ samples) for training and test. Besides, more recent AUC optimization methods and PU learning methods should be compared.

Thank you for the constructive suggestion. We performed the additional experiments on larger datasets. For each dataset, we used 100 positive and 9,000 unlabeled data in the training distribution and 9,000 unlabeled data in the test distribution for training. In addition, we included the recent PU learning method (PURA) [a] for comparison. Since it is designed for ordinary PU learning, it used PU data in the training distribution. Margin $\rho$ was selected from $\\{0.1,1,10\\}$ by validation data. The results are described in Table A of the anonymous URL. Our method outperformed the other methods. Since PURA does not consider distribution shift, it did not work well. We will include these results in the revised paper.

[a] Positive-Unlabeled Learning with Label Distribution Alignment, TPAMI'23

While the paper covers a broad range of related work, there are a few essential references that could provide additional context for the key contributions:

Thank you for sharing the papers. We briefly explain the difference between our work and these papers. The paper [1] proposed a PU learning method under class-prior shift. Unlike our method, it cannot deal with the covariate shift. The paper [2] proposed a PU learning method with label distribution consistency, but it cannot treat distribution shifts. The paper [3] proposed a PU learning method from noisy positive and unlabeled data. It cannot treat distribution shifts. In addition, all these methods are not designed for AUC maximization. We will include a more detailed discussion of these papers in Section 2.  

The paper draws on the importance weighting framework from covariate shift literature (e.g., Sakai, T. and Shimizu, N. Covariate shift adaptation on learning from positive and unlabeled data. In AAAI, 2019) but adapts it for AUC maximization (e.g., Kumagai, A., Iwata, T., Takahashi, H., Nishiyama, T., and Fujiwara, Y. AUC maximization under positive distribution shift. In NeurIPS, 2024). In general, the novelty might be somewhat incremental.

The novelty of our paper is mainly twofold. The first is to propose a novel and significant problem setting of AUC maximization under covariate shift with PU data. The second is to theoretically derive two novel AUC risk estimators to address this problem based on importance weighting, effectively utilizing the properties of AUC maximization and PU learning. Moreover, we also proposed to use the recently proposed dynamic importance weighting framework (Fang et al., 2020; 2023) for AUC maximization with PU data on complex models and data. As other reviewers acknowledged, we believe our method has sufficient technical novelty.

The method assumes a specific type of distribution shift (covariate shift), which may limit its applicability in scenarios with other types of shifts.

As you mentioned, our paper assumes the covariate shift. This is because the covariate shift is the most common and important shift in practice (He et al., 2023). Note that unsupervised distribution shift adaptation methods require some assumption about the shift type since no supervision is available in the test distribution. Since many papers at top conferences focus on the covariate shift only [b, c, d, e, f], we believe that our work has sufficient contributions.

[b] Test-time Adaptation for Regression by Subspace Alignment, ICLR'25

[c] Robust importance weighting for covariate shift, AISTATS'20

[d] Double-weighting for covariate shift adaptation, ICML'23

[e] Adapting to continuous covariate shift via online density ratio estimation, NeurIPS'23

[f] An information-theoretical approach to semi-supervised learning under covariate-shift, AISTATS'22

Meanwhile, we additionally evaluated our method under a class-prior shift, in which the class-prior changes, but the class-conditional density remains the same. Table C in the anonymous URL shows the results. Our method empirically worked well. This would be because our method does not depend on class-priors and thus is relatively robust against the class-prior shift. We will include these results.

Thank you for your positive and constructive comments. We will revise the paper according to your suggestion regarding the structure and notation. We added experimental results in the anonymous URL: https://anonymous.4open.science/r/icml25s-6C74/results.pdf

the benchmark datasets could be further improved: only small-scale datasets are used to validate the effectiveness.

We performed the additional experiments on larger datasets. Our method outperformed the others in Table A of the anonymous URL. Please see the answer to the first comment of Reviewer Gocn for details.

This work focuses on the covariate shift, i.e., the input distributions are different. However, as described in lines 315~328, the training data and test data are sampled under different class priors (categories of the images). A sampling strategy based on the inputs (e.g., style, noise, color) instead of the classes would be more reasonable.

Thank you for the constructive comment. As described in Line 317, we created the covariate shift by using the sampling strategy of the AISTATS paper of covariate shift adaptation (Aminian et al., 2022). Specifically, we first constructed positive and negative classes by partitioning the dataset's original classes into two groups. Then, we created $p_{{\rm tr}} ({\bf x}) \neq p_{{\rm te}} ({\bf x})$ by reversing the high- and low-density input regions based on the original classes between the training and testing. In our setting, training and test class-priors were the same ($\pi_{{\rm tr}} = \pi_{{\rm te}}$). Also, we used the epsilon dataset in Appendix E.3, where the covariate shift was created based on the inputs (the distance between the feature vectors) as in (Sakai & Shimizu, 2019). We would like to use the suggested strategy in future.

Details of the competitors (teAUC, trteAUC, UDAUC) are missing.

As described in the second paragraph of Section 5.2, the loss functions of teAUC and trteAUC are equivalent to Eq. (19) and Eq. (20) with $w({\bf x})=1$ for all ${\bf x}$ (i.e., they do not use importance weights), respectively. The loss function of UDAUC is a weighted sum of the loss of trAUC and the coral loss that is calculated from unlabeled training and test data. The coral loss is used to minimize the feature discrepancy. teAUC used positive training and unlabeled test data. trteAUC and UDAUC used positive training, unlabeled training data, and unlabeled test data as in our method. We will describe their specific loss functions in the revised paper.

Some results seem counter-intuitive. PAUC only achieves about 0.23~0.5 under most settings.

Please see our response to Reviewer 9PVZ's comment '(2) Why do the PAUC results in Table 1 perform particularly poorly.'

As $\pi$ varies from 0.01 to 0.1, the tasks become more manageable with more positive examples, but the test AUC drops for most methods.

AUC-based methods, except for PAUC, prefer small $\pi$. This is because they essentially use loss functions of the form, $\mathbb{E} _{{\bf x} ^{{\rm p}} \sim p ^{{\rm p}}} \mathbb{E} _{{\bf x} \sim p} \left[ f({\bf x} ^{{\rm p}}, {\bf x} ) \right]$, where $p ^{{\rm p}}$ and $p$ are positive and marginal densities. When $\pi$ is small, $p$ can be regarded as negative density $p ^{{\rm n}}$. In this case, the above loss function becomes the original AUC risk. Thus, these methods work well with small $\pi$. Since PAUC has a different form of loss, its trend was different. We will clarify this.

sensitive analysis on $\alpha$ should be provided.

We have performed the sensitive analysis on $\alpha$ in Appendix E.2. Although the tendency of the results varied across datasets, our method with $\alpha = 0.5$ worked well.

the learning rate is set to $10^{-4}$ for all methods, which might be suboptimal for some competitors.

In our preliminary experiments, we tested learning rates of $10^{-3}$ and $10^{-4}$ and confirmed no significant difference in performance. Based on this, we used $10^{-4}$ in this paper. We will mention this.

A key to removing the positive-positive loss in Eq. (15) is using a symmetric function... Is it possible to practically use other surrogate losses, such as hinge or square loss?

Yes, we can use non-symmetric losses, although the second term in Eq. (14) is not a constant as you mentioned. In this case, $\pi_{{\rm tr}}$ is necessary for training, but it can be estimated from PU data (Kumagai et al., 2024). Since the sigmoid function worked well in many PU learning and AUC maximization works, we also used it. We will discuss this in the revised paper.

How does the proposed method perform without a covariate shift?

We additionally evaluated our method when there were no shifts. Table B of the anonymous URL shows the results. Since there were no shifts, we compared trPU and trAUC, which do not consider shifts. Our method and trAUC showed comparable results, indicating that our method robustly works well when no shift exists. We will include this result.

We appreciate your positive comments on our paper.