Semi-supervised Multi-label Learning with Balanced Binary Angular Margin Loss

First of all, we are very grateful for your time and effort in reviewing this submission. We are encouraged that you agree with the contributions of our paper. Below are the responses to your comments.

Q1: Lack of quantitative analysis about the variance difference.

A1: Yes, adding quantitative analysis about the variance difference will greatly strengthen the paper. Therefore, we compare the variance difference between feature distributions (VDFD) of positive and negative samples of our S$^2$ML$^2$-BBAM during the training procedure with ones of CAP, SoftMatch and FlatMatch, as well as the ablation variant of our S$^2$ML$^2$-BBAM by replacing BBAM with BAM (“w/o BBAM”) on label 6 of VOC2012. The experimental results are summarized as follows:

From the results, the VDFD in the training procedure of our S$^2$ML$^2$-BBAM is much smaller than baselines, which demonstrates that S$^2$ML$^2$-BBAM can effectively balance the variance bias existing in current semi-supervised multi-label learning methods and obtain more fair performance. Otherwise, removing BBAM will increase the VDFD in the training procedure, and this shows the effectiveness of BBAM in balancing the variance bias.

Q2: Lack of experiments over benchmarks with more positive classes for each sample.

A2: We agree with you and add a new benchmark dataset, Animals with Attributes2 (AWA) [1], whose average number of positive classes is 30.78. Here are the experimental results on it:

In particular, our S$^2$ML$^2$-BBAM also performs better than baselines in most cases even when there are more positive classes for each sample.

Q3: Some minor typos.

A3: Thank you for your correction. We will update them and revise the manuscript.

Reference

[1] Lampert, C. H., H. Nickisch, S. Harmeling. Attribute-based classification for zero-shot visual object categorization. IEEE TPAMI, 36(3):453–465, 2013.

Thank you for reviewing our paper. We are encouraged that you agree with the novelty and contributions of our paper. Below are the answers to your questions.

Q1: Some efficiency experiments are expected to compare the real running time of the proposed method and other baselines. Does it limit the scalability of the proposed method for other datasets including load of samples or categories?

A1: Thank you for your suggestion, we compare the real running time of our S$^2$ML$^2$-BBAM with the current semi-supervised multi-label learning method CAP and two semi-supervised learning methods SoftMatch and FlatMatch over VOC2012 and Animals with Attributes2 (AWA) [1], whose numbers of samples are 5717 and 30337, numbers of categories 20 and 85, average numbers of positive classes per instance 1.46 and 30.78. We report the average running time per epoch by running 100 epochs as follows:

As shown in the table, the running time of our S$^2$ML$^2$-BBAM is competitive with CAP, and much less than SoftMatch and FlatMatch. We kindly argue that the Gaussian linear transformation, estimating label angle variances and negaive sampling are simple, their additional computational cost is very few and they only perform one time per epoch. Overall, the scalability of our proposed S$^2$ML$^2$-BBAM is well and it can be adapted to other datasets including load of samples or categories. We will discuss the additional computational cost and show efficiency evaluations in the next version.

Q2: The reliance on the Gaussian assumption for feature distribution could limit the method's effectiveness in cases where this assumption does not hold.

A2: Thank you for your discussion. The Gaussian can be considered as a general tool for many cases. Besides, the linear transformation characteristics of Gaussian lead to simple calculations. We will investigate other distributions such as von Mise-Fisher distribution and GMM in our future works.

Q3: Is it feasible to apply the idea of balancing feature angle distributions in other fields, such as imbalanced or long-tailed settings?

A3: Thank you for your discussion. We kindly argue that the idea of balancing feature angle distributions can be applied to imbalanced or long-tailed settings, because balancing feature angle distributions can increase the diversity of minor classes and decrease the diversity of major classes to some extent, then address the imbalance issue. We will exploit it in our future works.

Reference

[1] Lampert, C. H., H. Nickisch, S. Harmeling. Attribute-based classification for zero-shot visual object categorization. IEEE TPAMI, 36(3):453–465, 2013.

First, we are very grateful for your time and effort in reviewing this paper. Below are the responses to your questions and comments.

Q1: How does the negative sampling affect the performance of the proposed method? What happens if the proportion of positive and negative samples of each category is set to bigger or smaller values?

A1: Thank you for your comment. We kindly argue that negative sampling is employed to select high-quality negative samples for each label as mentioned in the introduction and is a widely-used trick in multi-label learning [1,2,3,4]. Besides, we also perform a sensitivity analysis of the proportion ($\eta$) of positive and negative samples of each category over VOC2012 with $\alpha$=5% to show the effect of negative sampling on the classification performance.

As shown in the table, the bigger or smaller values of $\eta$ will result in a worse classification performance. One possible reason is that the bigger $\eta$ will introduce many false negative samples and the smaller one will cause the number of negative samples is not enough. We will present the corresponding results and ablation study of negative sampling in the next version, and exploit the selection of $\eta$ in our future work.

Q2: What is the backbone for text benchmarks?

A2: Thank you for your correction. We use the bert-base-uncased as the text backbone and will clarify it in the next version.

Q3: The efficiency experiments may be performed.

A3: Thank you for your suggestion, we compare the real running time of our S$^2$ML$^2$-BBAM with the current semi-supervised multi-label learning method CAP and two semi-supervised learning methods SoftMatch and FlatMatch over VOC2012 and Animals with Attributes2 (AWA) [5], whose numbers of samples are 5717 and 30337, numbers of categories 20 and 85, average numbers of positive classes per instance 1.46 and 30.78. We report the average running time per epoch by running 100 epochs as follows:

As shown in the table, the running time of our S$^2$ML$^2$-BBAM is competitive with CAP, and much less than SoftMatch and FlatMatch. We kindly argue that the Gaussian linear transformation, estimating label angle variances and negaive sampling are simple, their additional computational cost is very few and they only perform one time per epoch. We will discuss the additional computational cost and show efficiency evaluations in the next version.

Reference

[1] Jiang, T., D. Wang, L. Sun, et al. Lightxml: Transformer with dynamic negative sampling for high-performance extreme multi-label text classification. In AAAI, pages 7987–7994. 2021.

[2] Dahiya, K., D. Saini, A. Mittal, et al. Deepxml: A deep extreme multi-label learning framework applied to short text documents. In WSDM, pages 31–39. 2021.

[3] Qaraei, M., R. Babbar. Meta-classifier free negative sampling for extreme multilabel classification. Machine Learning, pages 1–23, 2023.

[4] Liu, Weiwei, et al. The emerging trends of multi-label learning. IEEE TPAMI, 44(11): 7955-7974, 2021.

[5] Lampert, C. H., H. Nickisch, S. Harmeling. Attribute-based classification for zero-shot visual object categorization. IEEE TPAMI, 36(3):453–465, 2013.

First, we would like to thank you for your time and effort in reviewing our submission. Next, we would like to respond to the main concerns raised in the comments.

Q1: About the some unstandardized mathematical symbols

A1: Thank you for your correction. We change the ratio of labeled samples in training to $\pi$ and will update it as well as introduce a detailed explanation of mathematical symbols as you suggested in the next version.

Q2: About the computation complexity

A2: Thank you for your comment, we compare the real running time of our S$^2$ML$^2$-BBAM with CAP and SoftMatch and FlatMatch over VOC2012 and Animals with Attributes2 (AWA) [1], whose numbers of samples are 5717 and 30337, numbers of categories 20 and 85, average numbers of positive classes per instance 1.46 and 30.78. We report the average running time per epoch by running 100 epochs as follows. As shown in the table, the running time of our S$^2$ML$^2$-BBAM is competitive with CAP, and much less than SoftMatch and FlatMatch. We kindly argue that the Gaussian linear transformation, estimating label angle variances and negative sampling are simple, their additional computational cost is very few and they only perform one time per epoch. The next version will discuss the additional computational cost and show efficiency evaluations.

Q3: About the evaluation metrics

A3: Thanks for your advice. We add two new metrics, including Hamming Loss and One Error. Then, the metrics used to evaluate the performance of multi-label classification cover example-based metrics (mAP, Hamming Loss, One Error) and label-based metrics (Micro-F1 and Macro-F1) [2]. The following two tables report the experimental results of Hamming Loss and One Error. As shown in two tables, our S$^2$ML$^2$-BBAM also performs better than baselines in most cases. The corresponding results will be updated in the next version.

For Hamming Loss,

For One Error,

Q4: Only 3 comparison algorithms were involved in Table 2

A4: Thank you for your comment. We add SoftMatch and FlatMatch as baselines for text datasets, where Back-Translation (very time-consuming) is employed as the strong-augmentation method and none of the weak-augmentation methods is used following [3,4]. The corresponding results are reported in the following two tables. As shown in the two tables, our S$^2$ML$^2$-BBAM consistently outperforms two new baselines across all text benchmarks and metrics, demonstrating its effectiveness. We will update them in the next version.

For Ohsumed,

For AAPD,

Q5: Confusing description in 265-266

A5: Thank you for your correction. We want to say that the performance is insensitive to the rescaled norm $s$ when $s\in [10, 50]$, and the best result is obtained when $s=20$. We will clarify it in the next version.

Q6: Overlook label correlation

A6: Thank you very much. We agree that label correlation is important to multi-label learning and we should present the overlook of label correlation in Limitations. The label correlation may be helpful to generate high-quality pseudo-labels for unlabeled instances, such as label propagation based on labeled instances and label corrections, as well as regularization term based on label correlation for the classifier weights. We will exploit them in our future work.

Reference

[1] Attribute-based classification for zero-shot visual object categorization. IEEE TPAMI, 2013.

[2] A review on multi-label learning algorithms. IEEE TKDE, 2013.

[4] SoftMatch: Addressing the Quantity-Quality Tradeoff in Semi-supervised Learning. ICLR, 2023.