Can Class-Priors Help Single-Positive Multi-Label Learning?

Thank you for taking the time to review the paper and providing valuable feedback. I appreciate your efforts in ensuring the quality of the research. Regarding your concerns, I would like to provide the following explanations:

1. The warm-up strategy using AN solution seems necessary but lacks theoretical grounding. How sensitive is the class-prior estimation to the quality of the initial model predictions? Poor initial models might lead to unreliable prior estimates, creating a chicken-and-egg problem.

The warm-up strategy using AN loss is actually a well-established practice in the SPMLL, adopted by several classical algorithms [1-4]. To address your concern about sensitivity to initial model quality, we conducted an ablation study on VOC and COCO datasets, training the warm-up model for different numbers of epochs (1, 2, and 3) to simulate varying initial classifier quality:

The results demonstrate that even with a weak initial classifier (1 epoch, mAP of only 43.54% on VOC), CRISP achieves strong final performance (88.50%). The final performance remains stable across different warm-up qualities, with only minor improvements as the warm-up classifier gets stronger. This suggests that the class-prior estimation mechanism is sufficiently robust to work with imperfect initial predictions and progressively refine them during training.

[1] Label-aware global consistency formulti-label learning with single positive labels

[2] One positive label is sufficient: Single-positive multi-label learning with label enhancement.

[3] Large loss matters in weakly supervised multi-label classification

[4] revisiting pseudo label for single positive multi label learning

2. The related work section reads more like a literature survey than establishing clear connections to the proposed approach.

Thank you for this valuable feedback. We will revise the related work section in the next version to more clearly establish the connections between our proposed approach and existing methods.

3. Have the authors considered other class-prior estimation techniques beyond the threshold-based approach? Comparison with alternative strategies would strengthen the contribution.

We have actually compared our threshold-based class-prior estimation with an alternative strategy in our ablation study (Table 3). Specifically, we used the frequency of each labeled positive sample appearing in the validation set as the class prior. Our experimental results show that our approach outperforms the alternative strategy, demonstrating the effectiveness of our proposed method.

We hope that our revisions and additional clarifications have fully addressed your concerns. Please feel free to let us know if further explanations or additional analyses are needed.

Thank you for taking the time to review the paper and providing valuable feedback. I appreciate your efforts in ensuring the quality of the research. Regarding your concerns, I would like to provide the following explanations:

For Weaknesses:

1. the methods used for comparison are from several years ago, it would be better to make comparison with the latest papers on this topic.

Thank you for the valuable suggestions. We note that [5] (LL) has already been included as one of our baselines. In the revised version, we additionally include [1] and [3] as new baselines, and the comparison results are shown below. Our method achieves superior performance on most datasets, with only a slight gap compared to MIME on the CUB dataset. Regarding [2] and [4], both methods rely on CLIP-based pseudo-labeling, which introduces extra supervision information beyond single-positive labels. Direct comparison with our method may therefore be unfair. However, we will discuss these approaches in the Related Work section to provide a more comprehensive overview.

2. One thing confused me is the metrics utilized for demonstrating your experiment results and make comparisons. Why choose ranking loss? It would be better to add the commonly used metrics including mAP, F1, Recall, etc.

Thank you for the suggestion. In fact, we have already evaluated our method using multiple commonly used metrics, including mAP, Ranking Loss, Hamming Loss, One-Error, and Coverage, as shown in Tables 1, 2, 8, 9, 10, and 11. These metrics jointly assess different aspects of model performance, such as ranking quality, per-label accuracy, and top prediction correctness. We will clarify this in the revised version to avoid confusion.

We hope that our revisions and additional clarifications have fully addressed your concerns. Please feel free to let us know if further explanations or additional analyses are needed.

Thank you for taking the time to review the paper and providing valuable feedback. I appreciate your efforts in ensuring the quality of the research. Regarding your concerns, I would like to provide the following explanations:

1. l. 116: "ratio between the fraction of the total number of samples and that of positive labeled samples receiving scores above the threshold, thereby obtaining the class-prior probability of the j-th label": this was pretty unclear.

What we intended to express is: For label j, we calculate the class-prior $\pi_j$ as the ratio between the total number of samples and the number of samples with scores above threshold $\hat z$. For example, if we have 1000 samples in total and 300 samples have scores above $\hat z$, then the estimated class-prior $\pi_j = 300/1000 = 0.3$​. We will clarify this in the revision.

2. l. 123: empirical estimator : since this is estimating an expectation by an un-weighted sum, you need to describe the sampling of from for this estimator.

The samples $\boldsymbol{x}_i$ in our empirical estimator are independently and identically distributed (i.i.d.) samples drawn from the marginal distribution $p(\boldsymbol{x})$. We will add this clarification to make the estimator more rigorous.

3. eqs 6, 7: the class priors you are using for supervision can not be known in ground-truth. Are you in fact using the estimations as per Algorithm 1?

We confirm that the class priors used in Eqs. (6) and (7) are indeed the estimates obtained via Algorithm 1.

4. you never demonstrate that $\pi_i = p(y_i=1)$

While we have described $\pi_i$ as the class prior probability in the text, we acknowledge that we should have explicitly stated the mathematical relationship $\pi_i = p(y_i=1)$. We will add this explicit equation in the revised version.

5. eq 4: you should mention that this is binary cross-entropy. In general, your thresholded classifier approach only makes sense in contexts where the classifier's outputs can be interpreted as a confidence score for each class. A classifier trained on a hinge loss, for example, would not fit this framework.

We would like to clarify that our method uses L1 loss (absolute loss) as stated in Eq. (5). We agree that the loss should allow the classifier output to be interpreted as a confidence score, which can be achieved by applying a sigmoid function to the logits.

6. the "class-prior" vocabulary implies a Bayesian framework, but the notion of updating priors based on data is contradictory to the Bayesian viewpoint, where the prior is fixed a priori. Bayesian data-based updates are typically made on the posterior.

We clarify that in our setting the true class-prior probabilities are indeed fixed, consistent with the Bayesian framework. What we update during training is the estimate of the class-priors, not the true priors themselves. As the classifier improves, the estimated class-priors become more accurate, and this alternated optimization leads to better performance.

7. why do you split data differently for MLL and MLIC datasets?

For the MLIC datasets, the train/test splits are predefined, so we simply hold out a portion of the training set for validation. In contrast, the MLL datasets do not have predefined splits, so we manually split them into train/validation/test sets. This difference in splitting strategy does not affect the fairness of comparison.

8. Why are separate statistics being reported for MLL and MLIC (tables 6 and 7). I'd like to know table 6 and 7's reported statistics on both the MLIC and MLL datasets.

As mentioned above, the MLIC datasets are pre-defined image datasets, so we report their statistics in terms of #Training / #Validation / #Testing / #Classes. In contrast, the MLL datasets are tabular datasets without predefined splits, so we report their statistics as #Examples / #Features / #Classes.

9. Naively, we could estimate by looking at the data only and counting the number of times the label is positive and dividing by the total number of observations. Why is the proposed method, which is more of a pseudo-signal since it is dependent on the classifier, better? Empirical evidence here would be most convincing.

In SPMLL, each instance is associated with only one observed positive label, while the remaining labels are unobserved. As a result, a large portion of positive labels is missing, making it impossible to accurately estimate class priors by simply counting the observed positive labels and dividing by the total number of samples. To address this challenge, our method iteratively refines class-prior estimates by leveraging both the classifier outputs and the limited available labels.. Moreover, we provide empirical evidence in the ablation study (Table 3), where we used the frequency of labeled positive samples in the validation set as the class-prior (“CRISP-VAL”). The results show that our proposed method achieves better performance than this naive approach, because our estimator leverages both the classifier’s outputs and the available labels to iteratively refine the class-prior estimates.

10. l. 240: "Without the constraint of the class-priors, the predicted class-prior probability diverges from the true class-prior as epochs increase, significantly impacting the model’s performance." The claim that model performance is impacted by this phenomenon is not sufficiently supported. Predicted and true probabilities do not necessarily have to be close for the correct discrete decision to be made. The issue of accurate probability prediction is one of model calibration, which has its own specialized performance metrics for this explicit reason; standard performance metrics are often agnostic to calibration.

We will revise this statement in the next version. Our intention was to suggest that lack of class-prior alignment may be one factor contributing to poorer performance, rather than claiming a direct causal relationship. Nonetheless, our experiments clearly show that our method achieves better alignment between predicted and true class-priors.

11. Figure 1: how do you compute the predicted class-priors for these methods. Presumably, they do not all include explicit class-prior prediction strategies in their published methodologies.

For all methods, we compute the predicted class-priors as the average predicted confidence score for each class across all samples. Even though most baselines do not explicitly estimate class-priors, their model outputs can still be interpreted as confidence scores, so taking the mean over samples provides a consistent way to compare predicted class-priors across methods.

We hope that our revisions and additional clarifications have fully addressed your concerns. Please feel free to let us know if further explanations or additional analyses are needed.

Thank you for your continued feedback. We would like to provide the following clarifications:

If we assume that labels are masked uniformly at random in the train-set to build the SPMLL setting ...

Our method does not require the assumption that single-positive labels are sampled uniformly at random. Even under non-uniform selection of the single positive label, our class marginal probabilities estimation framework remains valid. On the other hand, even under the uniform sampling assumption, estimating the true class marginal probabilities by directly counting the frequency of observed labels in train-set is still biased.

This is because the probability $p(l ^ k = 1)$, i.e., label $k$ being selected as the single positive label, is not equal to $p(y ^ k = 1)$, but rather proportionally biased downward, due to the uniform sampling over the set of positive labels. We include the derivation below to illustrate this point:

Let $\boldsymbol{y} \in \{0,1\} ^ c$ be the binary label vector with c classes, and let $l ^ k = 1$ denote that class $k$ is selected as the observed single-positive label. If the positive label is sampled uniformly from the set $\{ j : y ^ j = 1 \}$, then for any $k$ such that $y ^ k = 1$, the probability that it is selected is:

$$p(l^k = 1 \mid \boldsymbol{y},y^k=1) = \frac{1}{\sum_{j=1}^c \mathbb{I}(y^j = 1)}$$

Then, marginalizing over $\boldsymbol{y}$, we get:

$$p(l^k = 1) = \int p(l^k = 1 \mid \boldsymbol{y}) p(\boldsymbol{y}) d\boldsymbol{y}$$

The term $p(l ^ k = 1 \mid \boldsymbol{y})$ can be decomposed as follows:

$$p(l ^ k = 1 \mid \boldsymbol{y}) = p(l ^k = 1 \mid \boldsymbol{y},y ^k=0)p(y ^k=0|\boldsymbol{y}) \\ + p(l ^k = 1 \mid \boldsymbol{y},y ^k=1)p(y ^k=1|\boldsymbol{y})$$

Given that $y ^ k = 0$ implies the $k$-th label cannot be selected as the single positive label, the term $p(l ^ k = 1 \mid \boldsymbol{y}, y ^ k = 0)$ becomes zero. Thus, the expression simplifies to:

$$p(l^k = 1 \mid \boldsymbol{y}) = p(l^k = 1 \mid \boldsymbol{y},y^k=1)p(y^k=1|\boldsymbol{y})$$

Substituting this into $p(l ^ k = 1) = \int p(l ^ k = 1 \mid \boldsymbol{y}) p(\boldsymbol{y}) d\boldsymbol{y}$：

$$p(l ^ k = 1) = \int p(l ^ k = 1 \mid \boldsymbol{y}) p(\boldsymbol{y}) d\boldsymbol{y} \\$$ $$= \int p(l ^k = 1 \mid \boldsymbol{y},y ^k=1)p(y ^k=1|\boldsymbol{y}) p(\boldsymbol{y}) d\boldsymbol{y} \\$$ $$= \int \frac{1}{\sum _{j=1} ^c \mathbb{I}(y ^j = 1)} \cdot p(\boldsymbol{y}|y ^k = 1) \cdot p(y ^k=1) d\boldsymbol{y} \\$$ $$= p(y ^k = 1) \cdot \mathbb{E} _{\boldsymbol{y} \sim p(\boldsymbol{y}|y ^k=1)} \left[ \frac{1}{\sum _{j=1}^c \mathbb{I}(y ^j = 1)} \right]$$

This gives:

$$p(l^k = 1) = \alpha \cdot p(y^k = 1)$$

where $\alpha = \mathbb{E} _ {\boldsymbol{y} \sim p(\boldsymbol{y}|y ^ k=1)} \left[ \frac{1}{\sum _ {j=1} ^ c \mathbb{I}(y ^ j = 1)} \right]$. Thus, the frequency of observing $l ^ k = 1$ under uniform sampling is a biased estimate of $p(y ^ k = 1)$, and must be re-scaled by $1/\alpha$ to recover an unbiased estimate. However, the scaling factor is itself nontrivial to estimate, which makes naive frequency-based methods on train-set unreliable in practice.

If there are enough samples, then your claim that naive measurement is impossible. If there are not enough samples, then your CRISP-VAL results when ...

1. Our core motivation lies in addressing the class imbalance challenge in SPMLL by proposing a general framework that combines a class marginal probability estimator with a tailored loss function. Both our method (CRISP) and the frequency-based estimator using the validation set (CRISP-VAL) achieve competitive performance, which validates the effectiveness of our framework regardless of the estimation strategy.

2. Regarding the concern that CRISP-VAL performs close to CRISP, we believe this is because we held out 20% of the training data as validation, which provides a relatively large and clean set to estimate the class marginal probabilities $p(y ^ j = 1)$. However, in real-world SPMLL scenarios, obtaining such a well-annotated validation set is costly and often unrealistic.

   To further demonstrate this, we conducted an additional experiment using only 5% of the training data as the validation set to estimate the class-priors. The results show a notable performance drop, confirming that frequency-based estimates degrade rapidly when fewer observed labels are available, while our method is robust.

These results further motivate the need for an accurate, data-efficient estimator like ours in low-annotation regimes.

We hope that our revisions and additional clarifications have fully addressed your concerns. Please feel free to let us know if further explanations or additional analyses are needed.

This mostly resolves my concern. I will raise my score to 4: Borderline Accept.

I have one final clarification to ask, related to the previous response: When you say you take 5% of the training data as validation set for the new version of CRISP-VAL results you show, are you using the pre-SPMLL-masked labels or the post-masked data.

I understand the point you make that sampling from the masked data biases frequency estimates, but it would be useful to see to what magnitude this bias occurs in practice.

Thank you for taking the time to review the paper and providing valuable feedback. I appreciate your efforts in ensuring the quality of the research. Regarding your concerns, I would like to provide the following explanations:

For weaknesses:

1. The effectiveness of the method appears to rely in part on the performance of the warm-up classifier; if the initial classifier is too weak, it may undermine the reliability of thresholding and lead to degraded practical performance.

To evaluate how much our method depends on the quality of the warm-up classifier, we conducted an additional ablation study on VOC and COCO. Specifically, we trained the warm-up model using the AN loss for 1, 2, and 3 epochs to simulate classifiers with varying initial quality. The results are shown below:

As shown, our method remains stable even when the warm-up classifier is relatively weak. This suggests that while the initial classifier provides a starting point for thresholding, the overall method is robust to its performance within a reasonable range.

2. Some recent baselines are missing, such as “Exploring Structured Semantic Prior for Multi-Label Recognition With Incomplete Labels.

Thank you for pointing this out. In the revised version, we have added new baselines, including MIME[1] and GR Loss[2], and the updated comparison results are shown in the table below. Our method outperforms these recent approaches on most datasets, with only a slight gap compared to the best method on CUB. Regarding “Exploring Structured Semantic Prior…” and other CLIP-based pseudo-labeling approaches, they introduce extra supervision information beyond single-positive labels, making direct comparison potentially unfair. However, we will explicitly discuss these methods in the Related Work section to provide a more comprehensive overview.

[1] Chen, Yanxi, et al. "Boosting single positive multi-label classification with generalized robust loss." Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence. 2024.

[2] Liu, Biao, et al. "Revisiting pseudo-label for single-positive multi-label learning." International Conference on Machine Learning. PMLR, 2023.

For Questions:

1. Will the inclusion of the additional absolute operator of Equation 7 compromise the unbias of the proposed risk estimator?

The absolute value is introduced to ensure the non-negativity of the risk and to better align the model-predicted class prior with the ground-truth class prior. This modification does not compromise the unbiasedness of the estimator, as formally shown in Theorem 4.2. Despite the introduction of the absolute operator, we provide a estimation error bound demonstrating that the empirical risk minimizer still converges to the optimal risk minimizer as $n \to \infty$. The bound has already accounted for this modification.

2. Why weren’t BOOSTLU and PLC compared on the MLL datasets?

As we explained in lines 224-227 of our paper, due to the inability to compute the loss function of PLC without data augmentation, we do not report the results of PLC on MLL datasets because data augmentation techniques are not suitable for the MLL datasets. Similarly, since the operations of BOOSTLU for CAM are not applicable to the tabular data in MLL datasets, its results are also not reported.

We hope that our revisions have addressed all of your concerns, but please let us know if there is anything else we can do to improve the manuscript. We would be happy to answer any additional questions or provide any further information you may need.