Make Small Data Great Again: Learning from Partially Annotated Data via Policy Gradient for Multi-Label Classification Tasks

Thank you very much for your thoughtful feedback and acknowledgment of our novelty and effectiveness.

(W.1) The motivation might be unreasonable. As stated in the abstract, "This task encounters challenges associated with a scarcity of positive annotations and severe label imbalance". These challenges are indeed problem to deal with, but in my opinion, the main challenges correspond to that the negative samples are unknown or we do not know which are negative samples among the remaining unlabeled samples. In other words, the main challenges are the same as PU learning.

A: We agree that we have the same challenge of unknown negative samples as PU learning. Such a challenge is commonly faced by partially annotated problems. Besides, we want to emphasize that solving multi-label classification tasks with partial annotation also encounters more severe difficulties than PU learning:

• positive-unknown imbalance;
• unknown prior positive-negative distribution or actual ratio of positive missing.

PU learning generally supposes an actual balanced positive-negatvie distribution or given the prior positive-negative distribution. Su et al. [1] puts forward doubts on the strong assumption and has explored PU learning from imbalanced data. Their setting in the PU learning scenario is the most related to us, and we have conducted experiments for comparison.

Our task setting can be considered as a more general form of PU learning and can be applied to more realistic task challenges, such as relation extraction and image classification in NLP and CV.

(W.2) For the two challenges mentioned in this paper, i.e., a scarcity of positive annotations and severe label imbalance, it is unclear how can the proposed PAPG method overcome them.

A: Our method addresses the above two challenges from two carefully designed mechanisms.

• Self-supervised data enhancement. -> Scarcity positive labels: We produce pseudo positives with consideration of both policy and value networks to guarantee pseudo positive classes with high confidence.
• Designing local rewards provided by value network and global rewards -> label imbalance: Our value network exploits data distribution with high-quality label augmentation; our policy network performs exploration guided by the value network. The global reward can encourage our policy network to predict a greater number of classes as TRUE, and the local rewards monitor the exploration of our policy network so as not to stray too far from the existing annotations.

(W.3) As stated in Introduction, "Consequently, many advanced methods of PU learning (Su et al., 2021; Acharya et al., 2022; Luo et al., 2021) cannot readily adapt to our multi-label settings". In my opinion, if we focus each label one by one, it is a PU learning problem. Thus, the partially annotated multi-label classifcation problem can be solved via binary relevance strategy where the base classifier is trained with any off-the-shelf PU learning methods.

A: Thanks for your thoughtful comments. Actually, we leverage binary relevance as the base of our framework. However, many advanced PU learning methods that suppose a given positive-negative distribution prior or assume a balanced positive-negative ratio are not adapted for our settings.

We want to emphasize the challenges of multi-label classification beyond those of PU learning:

• the severe positive-negative imbalance and missing labels exacerbate the imbalance and plague recognizing positives;
• unknown distribution prior.

Thus, our task setting can be considered as a more general form of PU learning, and our framework can be adapted for the PU learning setting. Therefore, we conducted experiments on the PU learning setting and compared to nnPU and ImbalancednnPI methods, a method for general PU learning and a specific method explored PU learning from imbalanced data. Compared to them, our framework also has superiority in the PU learning setting (demonstrated in Table 1 in experiment 1).

[1] Su et al., Positive-unlabeled learning from imbalanced data, IJCAI 2021.

(W.4) For multi-label classification, it is very important to model the correlations among labels. But it is unclear how can the proposed PAPG method model label correlations.

A: Thank you for pointing this out. Indeed, we have considered the label correlations in our challenge. However, in our setting, label correlations may bring correlation bias to the severe scarcity of positive labels. Therefore, we did not explicitly model the correlations in our model. A similar statement is also indicated in Huang et al. [2]: "Label correlation can be exploited to improve the performance of multi-label classifiers, and it is usually learned or calculated from the label matrix of the training data. But an inaccurate result of label correlation will be obtained if part of ground-truth labels is missing" and "Inaccurate decision function of a multi-label classifier will be made if the instances without one class label are simply treated as negative instances or a multi-label classifier tries to exploit label correlation from the incomplete label matrix directly".

Nevertheless, we do appreciate the suggestion and would like to explore the potential of leveraging the label correlations to enhance our framework in future work.

(W.5) The writing can be greatly improved. For example: (1) The title is inappropriate. (2) Errors exist in notations (especially the first paragraph in section 3.1). (3) What does RARL mean in the last sentence of this paper?

A: Thank you for your careful review. We have:

• revised the title to Learning from Partially Annotated Data via Policy Gradient for Multi-Label Classification Tasks;
• carefully refined the notations in Section 3.1 and Algorithm 1;
• fixed "RARL" to "PAPG".

Sorry about the typos; we have carefully reviewed our paper and fixed them in our revised version.

[2] Huang et al., Improving multi-label classification with missing labels by learning label-specific features, Information Sciences 2019.

We sincerely thank you for your constructive comments and acknowledgment of our work.

(W.1) The literature survey for weakly supervised learning in this paper is incomplete. The survey only covers some settings under multi-class single-label scenarios and misses the weak supervision under multi-label learning, such as missing-label learning. This miss is critical as the major novelty of this paper comes from proposing a method in this setting. The comparisions with such methods are also missing, so that the effectiveness of the proposed method is not well supported. The baselines are considerably inadequate, and the results on some famous multi-label classification datasets, such as NUS-WIDE dataset, are missing.

A: Thanks for your constructive comments.

For the literature survey for multi-label learning, we have noticed four settings of multi-label learning with weak supervision relevant to our setting.

• POL refers to partially observed labels, which supposes that partial annotations contain both positive and negative classes, such as [1].
• PPL refers to partially observed positive labels, which supposes that only positive classes are annotated but requires that there is at least one positive label per instance, such as [3].
• SPL refers to a single positive label, which supposes that each instance only has one positive label, such as [2].
• PML refers to partial multi-label learning, which supposes that each instance has a set of candidate labels where the number of positive labels is unknown, such as [4].

We focus on the partially observed setting with random ratio of positive class annotations and no negative class annotation, which is different from the above settings. It is worth specifically pointing out that we do not follow the PPL setting with promising at least one positive label per instance. Because in some tasks, such as relation extraction, the instance may not have any positive classes.

For comparisons with partially observed multi-label learning, we verify the advantage of our method in two domains:

• In the Nature Language Process domain, we conducted experiments in the relation extraction task and compared our method with some simple baselines, because there are no models designed for partially annotated settings in relation extraction.
• In the Computer Vision domain, we conducted an experiment in the multi-label image classification task. We have compared to previous work designed for POL setting [1] by re-running their official code on our datasets. We provide comparisons (MAP) with additional previous methods that provides code in the following Table.

For experiments on more multi-label classification datasets, such as NUS-WIDE dataset, to prove the effectiveness and generalizability of our framework, we conduct experiments with tasks in different domains, i.e., PU learning, NLP, and CV. Thus we only choose one representative multi-label image classification dataset. We will add more experiments on different datasets.

(W.2) The proposed method itself is simple and straightforward. Therefore, it would be better to further analyze the design choices of the proposed model to claim the impact of the proposed method. For example, why is the loss function used? Is there any better option for the authors to try for the loss function?

A: We have conducted an ablation and variant study in Section 4.4, which includes analysis of different reward designs and different training strategies. For example, we show the advantages of both local reward and global reward we designed, and we compared our framework with directly supervised self-training to show the benefits of RL module besides the benefits of pseudo positive classes augmentation, etc.

[1] Ben-Baruch et al., Multi-label classification with partial annotations using class-aware selective loss, CVPR 2022.

[2] Cole et al., Multi-label learning from single positive labels, CVPR 2021.

[3] Zhang et al., Simple and robust loss design for multi-label learning with missing labels, arXiv preprint arXiv:2112.07368, 2021.

[4] Xie et al., Partial Multi-Label Learning, AAAI 2018.

(Q1.) The novelty is debatable.

A: We want to underline our novelty:

• To the best of our knowledge, applying RL framework in a partially annotated multi-label classification problem is entirely new.
• We do not simply apply traditional RL methods for our setting. Instead, we specifically design novel reward functions and training strategies for the partially annotated challenge. The ablation study shows the effectiveness of our designs.
• We combine supervised learning with label augmentation and RL training to take advantage of both exploitation and exploration abilities, which is necessary when lacking positive labels.
• We demonstrate consistent improvements on three tasks in different domains, i.e., a PU learning setting, an NLP task, and a CV task, which shows the robustness and generalizability of our framework.

We sincerely appreciate you for your careful review and insightful comments. We provide detailed replies to your comments and hope we can resolve your major concerns.

(W.1) usually the algorithms contain the DRL methods will have a relatively higher variance on the performance. May I know what is the variance of your model's F1 and mAP in multi-label image classification?

How many time did you repeat your experiments? May I know what is the variance of your model's F1 and mAP in multi-label image classification?

A: Great question. We indeed considered two different scenarios to evaluate the robustness of our method.

• Different dataset versions: To simulate partial annotation, we randomly reserve a ratio of positive classes and treat other classes as unknown. To fully verify the effectiveness and stability of our model, we randomly constructed three versions of data sets (refer to Appendix B Table 6 for detailed results on DocRE). It can be seen that our method achieves consistent improvement in all the dataset versions.
• Different runs in the same dataset: For multi-label image classification, we repeat three runs in the same dataset. The standard deviations of F1/MAP scores of Table 2 in paper is as follows.

Thank you for pointing out. We have updated the above results in our revised paper.

(Q1) How large is the computation overhead in algorithm compared with the standard supervised learning?

A: We first want to clarify the relations between our methods and standard supervised learning methods.

• Our framework is applicable to be built upon any model structures with supervised learning.
• We used existing SOTA-supervised learning models as our policy and value networks.
• We supervised train our policy network.

In the training process: Our framework consists of a value network and a policy network, which leads to a doubling of parameters and training time compared to standard supervised learning.

In the inference process: We only leverage the policy network to decide $`TRUE`$ classes. Thus, the inference computation is equal to the standard supervised learning methods after geting a well-trained model.

Since our framework can be treated as a teacher-student architecture, we would like to explore the potential of a lightweight yet high-performance policy model in future work.

(Q2.) What will be the obstacles if you run this algorithm on a very large image dataset compared to the standard supervised learning? For example, 9 million partially annotated training images (OpenImages V6)?

A: Our framework is model-agnostic and acts as a plug-in strategy, so there are no problems to apply our framework for other datasets. However, there is no denying the fact that we utilize an extra value network to assist learning. It requires twice the storage and training time compared with standard supervised learning and thus needs more space and time computation on large datasets.

For the reasons we did not experiment on other datasets, the baseline model we chose for multi-label image classification is Ben-Baruch et al. [1], where they reported results on MS-COCO, OpenImages V6, and LVIS. But the code base provided by the author only contains the training code on MS-COCO and is not adapted to OpenImage V6.

Moreover, we conduct experiments with tasks in different fields, i.e., PU learning, NLP, and CV. Thus, we only choose one representive multi-label image classification dataset MS-COCO.

(Q3.) It would be better if you can show a figure reflecting the training process, e.g., x-axis: training steps, y-axis: accumulated reward or mAP.

A: Thanks for pointing out. Indeed, we have provided training curves during submission in Appendix B.1 of supplementary material. Specifically, in Figure 4, we display the reward and loss curves of our model in three annotation rations: 10%, 50%, and 100%. We would like to add more details if there are further concerns.

[1] Ben-Baruch et al., Multi-label classification with partial annotations using class-aware selective loss, CVPR 2022.

Thank you very much for your constructive comments and suggestions.

(W.1) Methods do not support assertions. This paper claims the local reward is the immediate reward in Sec. 1. and it provides immediate value estimation of each action in Sec. 3.2 However, these immediate rewards are summed up in dimension to get the final reward according to Eq. 3. It does not have immediate properties. It has the same frequency as the global reward. If this problem is not well explained, then this problem will be fatal.

A: Thank you for the careful comment. We want to clarify our rewards design by following aspects:

• As formulated in Section 3.2, we formalize the partially annotated multi-label prediction as a one-step MDP problem. In our reward design, the terms "local" and "global" are both used to characterize the "goodness" of a prediction given an input (i.e., a state-action pair). Therefore, they are both immediate rewards in the considered RL framework.
• By local reward, we refer to the reward assigned to the prediction of each class $y^c_i$. The underlying insight of such design has been stated in the Rewards part of paper: " Intuitively, the local rewards offer a preliminary yet instructive signal to guide the learning process in our PAPG framework. This signal inherits the *exploitation* aspect from the supervised loss training (as the value network is trained through supervised learning). Its purpose is to prevent the PAPG from engaging in excessively invalid exploratory behavior within the large action space, thereby enhancing the overall learning efficiency."
• By global reward, we refer to the reward assigned to the overall prediction $\mathbf{y}_i=[y^0_i, ..., y^{|C|}_i]$ of a data sample (i.e. a state $\mathbf{x}_i$), which "stimulate more comprehensive exploration during the learning process" and "mitigates distribution bias and encourages our policy network to predict a greater number of classes as TRUE.

Sorry about any confusion that we brought up here. We have revised our manuscript with further clarification in Introduction as the following paragraph: "Specifically, we combine the exploration ability of reinforcement learning and the exploitation ability of supervised learning by designing a policy network (as a multi-label classifier) learning with a policy-based RL algorithm and a value network (as a critic) trained with a supervised loss to provide local reward. Besides, beyond the local reward, we design a global reward assessing predictions of each instance, which contributes most to alleviating the imbalanced problem."

(W.1) I see no reason to use RL for this task.

A: We summarize the main motivation for using RL for our task:

• Inherent data balancing: As stated in Introduction, the traditional approach that uses weakly supervised learning (WSL) methods for multi-label classification tasks struggles with predicting of long-tail classes and leads to a performance drops significantly when the ratio of annotated positive classes decreases. Different from WSL, which focuses on exploiting the original incomplete data distribution, the exploration ability of RL has advantage of finding more positive classes when lacking positive annotations, and avoids the overfitting problem that supervised learning generally faces, especially when the observed data distribution is biased.
• Related Successful RL attempts: RL has been demonstrated as a successful learning approach in many weakly supervised settings (refer to Related Works Reinforcement Learning under Weak Supervision for details). Specifically, Luo et al. [1] has shown effective attempts by applying the RL framework to a simple PU learning setting. In our task, we inherit from a similar motivation to solve a more complicated and realistic task setting.
• Experimental Effectiveness: Empirically, as comprehensively presented in our Experiments Section, we show that our proposed RL framework, with careful reward designs and training strategy, has shown a consistent improvement across three tasks from different domains, especially when it comes to severe data imbalanced and extremely partial annotation problem. Refer to Table 1-3, Figure 1, Figure 3, for exemplar results.

[1] Luo et al., Pulns: Positive-unlabeled learning with effective negative sample selector. AAAI 2021.