A Closer Look to Positive-Unlabeled Learning from Fine-grained Perspectives: An Empirical Study

Q1. Could the author discuss whether the issue of estimating the positive sample distribution was considered in the research?

Thank you for your comment. While unknown positive class prior is indeed an important consideration in real-world applications, we would like to clarify our research positioning. Our work focuses on investigating the fundamental characteristics and techniques within existing PU learning paradigms, rather than addressing the prior estimation problem specifically. The assumption of known positive class prior is widely adopted in the PU learning literature.

Our work focuses on investigating the fundamental characteristics and techniques within existing PU learning paradigms, rather than addressing the prior estimation problem specifically. The assumption of known π is widely adopted in recent PU learning methods we compare against, as it allows for focused investigation of other critical aspects such as empirical risk design and pseudo-labeling strategies. Additionally, our proposed SAPU framework is orthogonal to prior estimation techniques, and can readily incorporate prior estimation methods like MPE/BBE as a preprocessing step. We believe combining SAPU with robust prior estimation represents a valuable direction for future work.

Q2. Several representative studies have not been discussed or compared. Could the author include a discussion on how these works relate to the research?

Thank you for your suggestion. [2] addresses the fundamental challenge of estimating the positive class prior $\pi$. As discussed in our response to the first question, this work is orthogonal to our research, since our method assumes $\pi$ is known to focus on empirical risk design and pseudo-labeling strategies.

[3] represents an extension of Dist-PU that we discussed in our disambiguation-free empirical risks family. [3] addresses the challenge of distribution mismatch between the positive labeled data and the positive instances within unlabeled data, which can violate the SCAR assumption. The label distribution alignment technique provides a way to handle such distribution shifts, making it a valuable enhancement to the Dist-PU.

[4] presents an alternative algorithmic method using random forests rather than neural networks we primarily focus on. We will expand the related work and discuss their relationship to our proposed method in the next version.

Q3. What is the specific motivation for combining SAPU and pseudo-labeling in the method? Which particular challenges in PU Learning does this combination address?

Thank you for your questions. The motivation for combining SAPU and pseudo-labeling lies in addressing the complementary limitations of existing disambiguation-free empirical risks and pseudo-labeling methods in PU learning. Our empirical analysis revealed that disambiguation-free empirical risks, while theoretically grounded, may suffer from gradient instability, particularly when the negative empirical risk $R_u^-(g) - \pi R_p^-(g)$ becomes negative. Conversely, pseudo-labeling methods can capture complex instance-level patterns but are highly dependent on pseudo-label quality, especially in early training stages where predictions are unreliable. By combining SAPU with pseudo-labeling into our GPU framework, we address both challenges simultaneously: SAPU provides stable set-level supervision through aggregate constraints that regularize the learning process, while pseudo-labeling enables fine-grained instance-level discrimination. This combination creates a more robust training objective.

Q4. Elaborating on the generalizability of the approach across related learning problems (e.g., UU Learning, CL Learning, Learning with New Class).

Thank you for this insightful question. The core innovation of our SAPU method lies in leveraging aggregate supervision through set-level label proportions, which naturally extends to various weakly supervised learning paradigms where similar distributional information is available. For instance, in UU learning, where we have two unlabeled datasets from different distributions, SAPU can be adapted by treating each unlabeled set with its estimated class proportion, formulating a cross-entropy loss that encourages the model to match these proportions at the set level. The mathematical foundation remains consistent: instead of $\pi$ representing the positive class prior in PU learning, we would have $\pi_1$ and $\pi_2$ representing the class distributions in the two unlabeled datasets.

For learning with new class, our GPU framework becomes particularly relevant. The pseudo-labeling component can be adapted to handle the discovery and classification of new classes by incorporating uncertainty estimation and cluster-based pseudo-labeling strategies. The set-aware supervision from SAPU can help maintain class proportion constraints even when new classes emerge.

Q5. Experimental design choices (backbones, epochs, and class definitions).

Thank you for your comment. We want to clarify that our experimental setup follows well-established practices in the PU learning community [1], which we cited and followed for consistency and fair comparison. The different epoch settings reflect established convergence patterns observed across the PU learning community for these standard benchmark datasets. F-MNIST typically reaches convergence faster due to its simpler feature space and lower intra-class variation, while CIFAR-10 and STL-10 require longer training periods to adequately learn the more complex decision boundaries present in natural images. These settings align with the training schedules used in the reference methods we compare against, ensuring that our experimental conditions provide fair and meaningful comparisons.

[1] Positive and unlabeled learning with controlled probability boundary fence.

Q6. Could the author elaborate on the process of setting and tuning hyperparameters (e.g., learning rates, weight decay)?

Thank you for your comment. We set learning rates as $5e-3$ for F-MNIST and $1e-4$ for others, and set weight decay as $1e-6$. We will add them in the next version.

Q1. SAPU is simply a straightforward application of aggregate supervision in PUL.

Thank you for this comment. While our SAPU is inspired by aggregate supervision, we argue that this is not a straightforward application but represents a principled innovation based on risk estimation theory. First, SAPU is built upon the theoretical foundation of disambiguation-free empirical risks, where we innovate in the risk estimation term by replacing the estimated negative risk term $\widehat{R}\_u^-(g) - \pi \widehat{R}\_p^-(g)$ (which causes negative risk issues in flexible models) with a theoretically grounded set-aware cross-entropy term that avoids optimization instabilities. Second, our subset partitioning strategy with formal variance analysis provides theoretical guarantees on proportion estimation accuracy, which is showed in Lemma 3.1 of our paper. More importantly, this work aims to establish a practical benchmark for PU learning by providing comprehensive empirical analysis and a unified framework. The integration framework GPU demonstrates how to effectively combine risk estimation with pseudo-labeling through careful design principles. So the novelty of our work lies not just in individual components but in establishing methodological principles that can guide future PU learning research.

Q2. Although it indicates that certain tricks are effective, it does not explain why.

Thank you for your suggestions.

Mixup proves consistently beneficial because it addresses the fundamental challenge of decision boundary uncertainty in PU learning. By creating synthetic samples through convex combinations, mixup naturally smooths the decision boundaries in regions. This is particularly crucial in PU learning where the model must distinguish between true negatives and mislabeled positives within the unlabeled set.

The counterintuitive phenomenon of performance degradation with moving average techniques primarily stems from the unstable nature of pseudo-labels in PU learning. Unlike traditional semi-supervised learning where unlabeled data contains truly unlabeled instances, PU learning involves mislabeled negative samples, making historical predictions unreliable. The self-training process generates systematic biases, and moving average perpetuates rather than corrects these biases. On the other hand, moving average techniques may suppress the model's ability to rapidly adapt within the feature space to distinguish between positive and negative samples. Furthermore, the momentum parameter requires careful tuning, which significantly increases the experimental cost for hyperparameter optimization.

Q3. This article lacks a detailed introduction to aggregate supervision.

Thank you for your comment. Aggregate supervision refers to learning from group-level constraints rather than instance-level labels. In traditional supervised learning, we have individual labels $y_i$ for each instance $x_i$. In aggregate supervision, we only know aggregate statistics about groups of instances, such as the proportion of positive instances in a set. Our method leverages this principle by treating the unlabeled dataset $D_u$ with class prior $\pi$ as aggregate supervision. Instead of trying to predict individual labels, we divide $D_u$ into subsets $\\{\mathcal{S}_i\\}\_{i=1}^S$ and enforce that each subset's predicted positive proportion approximates $\pi$. This strategy avoids the challenging individual pseudo-labeling problem and provides a natural regularization for overfitting. We will add the explanation about aggregate supervision in the next version.

Q4. Why does the choice of loss function in Table 4 have such a significant impact? How should we choose an appropriate loss function?

Thank you for these comments. The significant impact of loss function on the performance of SAPU reflects a fundamental mathematical incompatibility rather than algorithmic deficiency. Our SAPU employs cross-entropy loss for the set-level supervision term (Eq.9-10), which expects continuous probability outputs and relies on smooth gradient flow for effective optimization. In contrast, double-hinge loss is piecewise linear and promotes discrete, margin-based decisions through its sharp transitions at $z=1$ and $z=0$. This creates conflicting optimization dynamics where the set-aware component pulls toward smooth probabilistic outputs while the instance-level double-hinge component pushes toward sharp margin-based decisions. It explains why SAPU shows sensitivity to loss function compared to traditional disambiguation-free empirical risks (e.g., uPU, nnPU).

Based on the above analysis, we can summarize the following guiding principles:

(1) The smooth, differentiable losses (such as sigmoid, logistic loss) achieve better compatibility with the set-aware architecture of SAPU, while non-smooth losses (such as double-hinge, ramp loss) align better with traditional point-wise optimization methods.

(2) Convex losses generally provide better optimization guarantees.

(3) Simple datasets (e.g., F-MNIST) benefit from smooth losses, enabling fine-grained optimization, while complex datasets (e.g., CIFAR-10) may require losses with stronger regularization properties.

Thank you for your rebuttal. You have addressed most of my concerns. But I still believe that more larger and complex datasets should be added.

Q1. Why is the P$^3$Mix classified as a "hard pseudo-labeling" method? This paper fails to provide a clear classification standard for the distinction between "soft pseudo-labeling" and "hard pseudo-labeling."

Thank you for your comments. The P$^3$Mix method indeed falls under the category of "hard pseudo-labels." While mixup techniques inherently generate soft labels, our classification criterion is based on the final form of the labels used for training, rather than the intermediate processing steps. After applying mixup, P$^3$Mix still uses a hard threshold to generate binary pseudo-labels for the final training objective. To address the classification confusion, we formally define the distinction between hard and soft pseudo-labels as follows: pseudo-labels used for training are considered hard pseudo-labels if they are discrete binary values {$-1, +1$}, and soft pseudo-labels if they are continuous probability values $[0, 1]$. Based on this criterion, P$^3$Mix is classified as a hard pseudo-label method because its final training objective uses thresholded binary labels, even though the mixup process generates soft labels. We will clarify this definition in the revised version.

Q2. Regarding Table 4: a) Why SAPU consistently shows relatively poorer results with double-hinge loss? b) Different loss functions exhibit varying performance across different datasets. How can this phenomenon be explained? Are there general selection guidelines available?

Thank you for your suggestions. First, SAPU's underperformance with double-hinge loss stems from a fundamental structural incompatibility between our set-aware mechanism and the mathematical properties of loss function. Our SAPU employs cross-entropy loss for the set-level supervision term (Eq.9-10), which expects continuous probability outputs and relies on smooth gradient flow for effective optimization. In contrast, double-hinge loss is piecewise linear and promotes discrete, margin-based decisions through its sharp transitions at $z=1$ and $z=0$. This creates conflicting optimization dynamics where the set-aware component pulls toward smooth probabilistic outputs while the instance-level double-hinge component pushes toward sharp margin-based decisions. Additionally, our empirical analysis confirms that smooth, differentiable losses (sigmoid, logistic) achieve better compatibility with SAPU's set-aware architecture, while non-smooth losses (double-hinge, ramp) align better with traditional point-wise optimization methods.

Q3. The description about class-specific adaptive thresholding is unclear in lines 210-214.

Thank you for your comments. Unlike FlexMatch where $\tau$ is fixed at 0.95, our method dynamically adjusts both $\tau_p$ and $\tau_n$ based on the historical performance of pseudo-label quality. We first compute the accuracy rates for positive and negative pseudo-labels. Specifically, the accuracy of positive pseudo-labels at epoch t $\text{Acc}_p^{(t)}$ is the ratio of the number of correctly predicted positive pseudo-labels to the total number of positive pseudo-labels (, and similarly, the accuracy of negative pseudo-labels $\text{Acc}_n^{(t)}$ is the ratio of the number of correctly predicted negative pseudo-labels to the total number of negative pseudo-labels). Then the confidence ratios $C_p^{(t)}$ and $C_n^{(t)}$ are calculated as:

$C_p^{(t)} = \frac{\text{Acc}_p^{(t)}}{\max(\text{Acc}_p^{(1 :t-1)})} , \quad \quad C_n^{(t)} = \frac{\text{Acc}_n^{(t)}}{\max(\text{Acc}_n^{(1:t-1)})}$

where $\max(\text{Acc}_p^{(1:t-1)})$ represents the highest positive pseudo-label accuracy achieved in previous epochs. The thresholds are then updated: $\tau_p = \tau_p \times C_p^{(t)} , \quad \quad \tau_n = \tau_n \times C_n^{(t)}$ This mechanism provide adaptive confidence requirements that evolve with the model's improving discrimination capability.

Q4. Minor Issues.

Thank you for your corrections. We will revise them in the next version.

Q1. When the SCAR assumption is violated, are the methods in this work still applicable?

Thank you for your valuable comment. Unlike traditional PU methods that heavily rely on instance-level SCAR assumptions, SAPU operates through set-level supervision and the property of the positive sample selection mechanism being relatively independent of the feature distribution, which provides inherent regularization against selection bias. Therefore, when the SCAR assumption is violated, our method demonstrates certain robustness due to its aggregate supervision design. However, we acknowledge that when the SCAR assumption is severely violated, additional importance weighting or bias correction techniques are required. In the future, we plan to extend our work to develop adaptive strategies to handle SCAR violations.

Q2. Theorem 3.2 lacks deeper theoretical analysis.

Thank you for your valuable suggestion. We would like to clarify that our theoretical analysis does provide several important contributions beyond basic convergence. Theorem 3.2 establishes finite-sample bounds showing that the excess risk $R(\hat{g}\_{\text{SAPU}}) - R(g^*)$ converges at rates $O\left(\sqrt{\frac{\log(1/\delta)}{n_p}}\right) + O\left(\sqrt{\frac{\log(1/\delta)}{n_s}}\right) + L_\ell \cdot O\left(\sqrt{\frac{\pi(1-\pi)\log(1/\delta)}{S}}\right)$, which explicitly characterizes how the performance depends on the number of positive samples ($n_p$), number of subsets ($n_s$), subset size ($S$), and the positive class prior ($\pi$).

Additionally, Lemma 3.1 provides precise characterization of the deviation bounds for our set-aware method, showing that when $S \geq \frac{3\pi(1-\pi)\log(2/\delta)}{2\epsilon^2}$, we achieve $|\hat{\pi}_j-\pi|\le \epsilon$ with high probability, which directly informs the practical choice of subset size.

The theoretical analysis we provide serves to establish the soundness and risk-consistency of our SAPU method, which is sufficient to support our empirical findings and validate our proposed GPU framework.

Q3. Why does the incorporation of moving average techniques lead to performance degradation?

Thank you for your comment. This counterintuitive phenomenon of performance degradation with moving average techniques primarily stems from the unstable nature of pseudo-labels in PU learning. Unlike traditional semi-supervised learning where unlabeled data contains truly unlabeled instances, PU learning involves mislabeled negative samples, making historical predictions unreliable. The self-training process generates systematic biases, and moving average perpetuates rather than corrects these biases. On the other hand, moving average techniques may suppress the model's ability to rapidly adapt within the feature space to distinguish between positive and negative samples. Furthermore, the momentum parameter requires careful tuning, which significantly increases the experimental cost for hyperparameter optimization.

Q4. The GPU framework's performance improvements over existing methods remain within statistical error margins in most cases.

Thank you for this question. Regarding the limited performance improvements of the GPU framework, we acknowledge that compared to state-of-the-art methods, GPU's improvements are modest. This reflects several factors: First, strong baseline methods like PUL-CPBF and HolisticPU employ sophisticated techniques (ensemble learning, meta-learning) that are orthogonal to our contributions; Second, our focus is on fundamental technique analysis and principled integration, rather than achieving performance maximization through complex engineering; Third, GPU provides a general framework that can integrate future advances, while specialized methods may not generalize well. In the future, we will improve upon this by conducting experiments on more challenging real-world datasets and demonstrating GPU's scalability through the integration of ensemble techniques.