AUC Maximization under Positive Distribution Shift

Thank you for your positive comments and constructive feedback.

The effect of the proposed methods on MINST and Fashion MINST datasets is not significant, which is inconsistent with those on the other datasets. The authors don’t give any explanation.

As described in Line 279, since MNIST and Fashion MNIST are relatively simple data, the performances of our method and PURR that does not consider class-imbalance might be similar. More specifically, if the supports of positive and negative conditional densities are separated (i.e., data are simple), both losses of positive and negative data can be minimized without conflict between them, even when there is a class imbalance. Thus, class imbalance might not be a severe problem in this case. In contrast, when positive and negative-conditional densities overlap (i.e., complex data), minimizing losses for positive data located in the overlapped region causes to increase losses for negative data in the same region. Thus, when there is a class imbalance, positive data that are few can often be ignored. Therefore, in this case, class imbalance becomes a more severe problem. Since SVHN and CIFAR10 are more complex than MNIST and Fashion MNIST, our method worked well. We will clarify this.

The authors do not fully compare their method with the latest ones.

Thank you for sharing relevant works. We additionally compared our method with the method [a] in your comment (PURDA). The results are described in Table 5 of the PDF file in the global response. PURDA used positive and unlabeled data in the training distribution in this experiment since it is designed for ordinary PU learning. Margin $\rho$ is selected from $\\{0.1,1,10\\}$ by validation data. Our method performed better than PURDA. This is because PURDA does not consider the distribution shift. All the methods in your comment do not consider the distribution shift and, thus, are unsuitable for our setting. In the final version, we will include this result and add all related works in your comments in Section 2.

[a] Positive-Unlabeled Learning with Label Distribution Alignment. (TPAMI 2023)

All theoretical derivations are only based on the sigmoid surrogate loss. As far as I know, square loss is also popular. Can the theoretical results extend to the other losses?

This is an insightful question. As long as we use symmetric losses (i.e., loss $l$ satisfying $l(z)+l(-z)=K$ for any $z \in \mathbb{R}$ and $K$ is a constant) [6], we can derive the same final loss function in Eq. 13. This is because the second term in Eq. 10 becomes constant, and thus, we can ignore it. We note that symmetric losses include a wide range of losses such as sigmoid, ramp, unhinged, and zero-one losses, although the squared loss is not included [6]. We will include this discussion in the final version.

There are some typos.

Thank you for pointing out the typos. We will again carefully proofread and prepare the final version.

Thank you for your positive comments and constructive feedback.

How should the practitioner choose classification threshold after training their classifiers using your method?

Thank you for the insightful question. In practical use, there are many situations where it is beneficial just to be able to sort data in score order. For example, in anomaly detection, experts or operators can check data with high scores within the cost they can spend or until anomalous data do not appear. In disease diagnosis, patients with higher scores can be given priority for detailed examination. In recommendation systems, products can be presented to users in order of score.

However, when a classification threshold is required, we can use the estimated class-prior in the test distribution to determine the threshold. Specifically, we first extract negative data in unlabeled data from the training distribution by applying (off-the-shelf) PU learning to PU data in the training distribution. Since PU data and the class-prior in the training distribution are available in our setting, we can perform it. Next, since the negative distribution does not change in our setting, the extracted negative data can be regarded as negative data in the test distribution. We can estimate the class-prior in the test distribution by applying existing class-prior estimation methods [40, 59, 14] to the extracted negative and unlabeled data in the test distribution. The threshold can be set to the top $N_{{\rm te}} \pi_{{\rm te}}^{{\rm est}}$-th score of unlabeled data, where $\pi_{{\rm te}}^{{\rm est}}$ is the estimated class-prior. We will include this discussion.

Thank you for your insightful comments and constructive feedback.

Unlike the existing study [15, 42], the negative distribution is not considered.

The method in [15] assumes the positive distribution shift as in our method. Thus, it does not consider the negative distribution change. The method in [42] assumes the covariate shift, i.e., $p _{{\rm te}} (x) \neq p _{{\rm tr}} (x)$ but $p _{{\rm te}}(y|x) = p _{{\rm tr}} (y|x)$. Although the covariate shift has been traditionally studied, the assumption of $p _{{\rm te}}(y|x) = p _{{\rm tr}} (y|x)$ is often restrictive in practice. One of the main contributions of our work is to highlight an important problem (both class imbalance and positive distribution shift occur) that has been overlooked despite its many applications, as described in Section 1.

The extension of the proposed method is discussed but not evaluated in the experiments.

We have already evaluated the extension of our method (Eq. 16) in Table 6 of Section C.4. The extension of our method ($\alpha=0.999$) slightly tended to perform better than our original method ($\alpha=1.0$) by using additional labeled negative data.

Is the proposed loss function unbiased to its supervised counterpart?

Yes. As long as the assumption of positive distribution shift (Eq. 7) is satisfied, the derived AUC in Eq. 12 is equivalent to the original AUC in Eq. 8.

It would be valuable if there were discussion or experimental results showing the effect of the number of unlabeled data from the test distribution.

Thank you for the constructive comment. We additionally evaluate our method with a small number of unlabeled data from the test distribution. The results are described in Table 3 of the PDF file in the global response. As expected, the performance of our method tended to increase as the number of unlabeled data $N_{{\rm te}}$ increased. Nevertheless, our method tends to outperform puAUC, which does not use unlabeled test data, even when $N _{{\rm te}}=500$. Since many unlabeled data are often easy to collect, we believe that our method is useful in practice. We will include these results in the final version.

According to the literature, the non-negative risk estimator plays a crucial role in training deep neural networks. However, the proposed method does not mention the non-negativity of the risk estimator.

Thank you for the insightful comment. In our preliminary experiment, we evaluated our method with the non-negative loss correction (Ours w/ nn), which prevents the empirical loss from being smaller than zero. However, the results with and without the loss correction (Ours w/ nn and Ours) were almost identical, as described in Table 4 of the PDF file. Thus, we proposed the current simpler loss function.

We describe the non-negative correction used in this experiment. For clarity, we consider to minimize the AUC risk, $R _{\rho} (s) := \mathbb{E} _{{\bf x} ^{{\rm p}} \sim p ^{{\rm p}} _{{\rm te}} ({\bf x} )} \mathbb{E} _{{\bf x} ^{{\rm n}} \sim p ^{{\rm n}} _{{\rm te}} ({\bf x} )} [ f({\bf x} ^{{\rm n}}, {\bf x} ^{{\rm p}}) ]$, which is equivalent to maximize the AUC in Eq. 8 since  ${\rm AUC} _{\rho} (s) = 1 - R _{\rho} (s)$.  The minimum value of this risk is zero. Then, the corresponding loss for Eq. 13 becomes ${\cal L} _{{\rm risk}}(s) :=  \mathbb{E} _{{\bf x} \sim p _{{\rm te}} ({\bf x}) } \mathbb{E} _{{\bar {\bf x} } \sim p _{{\rm tr} } ({\bf x}) } [ f({\bar {\bf x}} ,{\bf x}) ] - \pi _{{\rm tr} } \mathbb{E} _{{\bf x} \sim p _{{\rm te}} ({\bf x}) } \mathbb{E} _{{\bf x} ^{{\rm p}} \sim p ^{{\rm p}} _{{\rm tr}} ({\bf x}) } [ f({\bf x} ^{{\rm p}} ,{\bf x}) ]$. Since this loss is derived from the AUC risk, $\mathbb{E} _{{\bf x} \sim p _{{\rm te}} ({\bf x} )} \mathbb{E} _{{\bf x} ^{{\rm n}} \sim p ^{{\rm n}} _{{\rm te}} ({\bf x} )} [ f({\bf x} ^{{\rm n}}, {\bf x}) ]$, it should not take negative values. Thus, to prevent negative values of its empirical estimate ${\hat {\cal L} _{{\rm risk}}} (s)$, we used the loss with the absolute value function $|{\hat {\cal L} _{{\rm risk}}} (s)|$ for the optimization. This correction is successfully used in PU learning [15] or other weakly supervised learning [a].

By the way, if class-prior $\pi _{{\rm te}}$ is known, we can derive a tighter lower bound on the AUC risk. Specifically, we can obtain $\mathbb{E} _{{\bf x} \sim p _{{\rm te}} ({\bf x} )} \mathbb{E} _{{\bf x} ^{{\rm n}} \sim p ^{{\rm n}} _{{\rm te}} ({\bf x} )} [ f({\bf x} ^{{\rm n}}, {\bf x}) ] \geq (1-\pi _{{\rm te}})/2$. Here, we used the definition of $p _{{\rm te}} ({\bf x} )$ and the fact that the AUC risk between the same densities is $1/2$ as described in Lines 163--164. By substituting Eq. 11 for this, we can obtain ${\cal L} _{{\rm risk}}(s) \geq (1-\pi _{{\rm te}})(1-\pi _{{\rm tr}})/2 =: b > 0$. As a result, we can use $| {\hat {\cal L} _{{\rm risk}}} (s) - b| + b$ for the optimization. Our method with this correction (Our w/ b) tended to enhance the performance of it without the correction (Ours) in Table 3. However, note that Ours has the strong advantage of not requiring the class-prior and performed better than existing methods. We will include this result in the final version.

[a] Lu, Nan, et al. "Mitigating overfitting in supervised classification from two unlabeled datasets: A consistent risk correction approach." AISTATS2020.

Regarding the Extension in Section 4.4, it would be nice to cite the existing work.

Thank you for the suggestion. Since the extension in Section 4.4 uses positive, negative, and unlabeled data in the training distribution, it is related to semi-supervised learning. A few studies [41,b] especially use the PU learning for semi-supervised learning. However, semi-supervised learning does not consider the distribution shift. The final version will include a more detailed discussion and citations.

[b] Sakai et al. Semi-supervised classification based on classification from positive and unlabeled data. In ICML2017

Thank you for your response.

We investigated the learning process for the training loss, validation loss, and test AUC. Here, we used ${\hat {\cal L} _{{\rm risk}}} (s)$ for the loss. Note that the validation loss is the value of ${\hat {\cal L} _{{\rm risk}}} (s)$ calculated with validation data (PU data in the training distribution and U data in the test distribution).

The training loss of our method became smaller than the lower bound $b$ as learning progressed and took negative on simple datasets (MNIST and Fashion MNIST). The validation loss (test AUC) tended to decrease (increase) initially but gradually increased (decreased) or stopped improving as learning progressed. These trends are consistent with ordinary PU learning, such as the study [22].

However, the validation loss and test AUC were well correlated, and our method was able to select good models by using early-stopping with the validation loss. This is one of the reasons for the good performance of our method without the loss correction. Early-stopping was effective in preventing overfitting in the PU learning context. Note that all methods in our experiments used early-stopping.

We will include this result and discussion in the final version.

Thank you for your insightful comments and constructive feedback.

An expansion to include various metrics, such as F-1 and G-mean of TPR and TNR, which are also relevant for imbalanced data classification, could enrich this paper.

Thank you for the suggestion. We agree that extensions to maximize other metrics such as F-1 could enrich our paper. However, AUC is the most representative metric for imbalanced data, and many papers (including top conference and journal papers such as ICLR, AAAI, ICCV, ICDM, NeurIPS, and TPAMI) focus solely on AUC maximization [12, 29, 41, 54, 55, 57, 58, 60, 62]. We, therefore, believe that the current proposal is still worthy of publication. Extending our problem setting to maximize other metrics is an interesting future work.

As a supplement, we evaluated F1 scores of each method. The results are described in Table 1 of the PDF file in the global response. Our method also outperformed the others even when it maximizes the AUC. This result may imply that AUC maximization can also help improve other evaluation metrics for imbalanced data. We will include this result and the above discussion (future work) in the final version.

the experimental validation is somewhat restricted, utilizing only four datasets, all of which are image datasets. A more comprehensive evaluation using a broader range of datasets is necessary to fully assess the proposed loss function's effectiveness.

Thank you for the constructive comment. We additionally evaluate our method with two tabular datasets (Hospital Readmission and Hypertension [a]). In Hospital Readmission, the task is to predict the 30-day readmission of diabetic hospital patients. In Hypertension, the task is a hypertension diagnosis for high-risk age. Both datasets are widely used for distribution shift adaptation studies [a]. We used positive-shifted data to construct the positive distribution shift. The experimental settings, such as the number of data, are the same as the submitted paper. The results are described in Table 2 of the PDF file. Our method outperformed the others. These results show that our method works well in tabular datasets. We will include these results in the final version.

[a] Gardner et al. "Benchmarking distribution shift in tabular data with tableshift." NeurIPS2023.

Q1: Please specify the scenarios where both class imbalance and positive distribution shift occur. Providing detailed examples will help readers grasp the practical significance of this research problem.

We have described the motivating examples (cyber security, medical diagnosis, and visual inspection) where both class imbalance and positive distribution shifts occur in the first and second paragraphs of Section 1. For example, in cyber security, malicious data (positive data) are much smaller than benign data (negative data). In addition, malicious adversaries often change their attacks (malicious data) to bypass detection systems, while benign data do not change much [9, 15, 63]. Thus, our problem setting is suitable for this application.

Q3: AUC maximization can be implemented not just as a loss function for neural networks, but across various machine learning methods. Why did you choose to focus on proposing a loss function?

We focus on the loss function because it has a high degree of generality: it can be combined with any (differentiable) models such as linear models, kernel models, and neural networks, as described in Lines 66--67. This characteristic is beneficial in practice because the available computing resources vary depending on the application site. For example, when resources such as GPUs are scarce, which is often the case, our loss function can be used with lightweight models such as liner models. When ample resources are available, huge models such as large neural networks can also be used.

Q4: The reviewer is not convinced that Lines 4-5 in Algorithm 1 sufficiently demonstrate the training process. There needs to be a more detailed and mathematical explanation of how model parameters are updated using the proposed loss function.

Sorry for the lack of explanation. We describe a more detailed and mathematical explanation of Lines 4 and 5 in Algorithm 1 below:

• Let $\\{ {\bar x} _{{\rm tr},m} ^{\rm p} \\} _{m=1}^{P}$ be sampled positive data from $X _{{\rm tr}} ^{\rm p}$ and $\\{ {\bar x} _{{\rm tr},m} \\} _{m=1} ^{M _{{\rm tr}}} \cup \\{ {\bar x} _{{\rm te},m} \\} _{m=1} ^{M _{{\rm te}}}$ be sample unlabeled data from $X _{{\rm tr}} \cup X _{{\rm te}}$. Then, in Line 4, we calculate the loss in Eq. 14 on the sampled data. That is, ${\hat {\mathcal L}} _{{\rm sampled}} (\theta) := - \frac{1}{M _{{\rm te}} M _{{\rm tr}}} \sum _{n,m=1} ^{M _{{\rm te}}, M _{{\rm tr}}} f({\bar {\bf x}} _{{\rm te}, n}, {\bar {\bf x}} _{{\rm tr}, m}) + \frac{\pi _{{\rm tr}}}{M _{{\rm te}} P} \sum _{n,m=1} ^{M _{{\rm te}}, P } f({\bar {\bf x}} _{{\rm te}, n}, {\bar {\bf x}} _{{\rm tr}, m} ^{{\rm p}})$, where $\theta$ is parameters of score function $s$.

• In Line 5, we update $\theta$ by using a stochastic gradient descent. That is, $\theta \leftarrow \theta - \mu \frac{\partial {\hat {\mathcal L}} _{{\rm sampled}}}{\partial \theta} (\theta)$, where $\mu \in \mathbb{R} _{>0}$ is a learning rate.

We will add this explanation in the final version.

L2: For extremely imbalanced cases, training difficulties are likely to arise. It is necessary to address how the proposed loss function can be effectively minimized in these scenarios.

As you mentioned, highly imbalanced data will be difficult to learn for all methods, including ours. This is due to the difficulty in extracting positive data information from unlabeled data. However, in Table 4 of Section C.2, we confirmed that our method outperformed the others when small class-priors on the training distribution (e.g., $\pi _{{\rm tr}}=0.01$) are used.