Multi-Instance Partial-Label Learning with Margin Adjustment

Thank you for your thorough review and positive feedback on our paper. We are pleased that you found the paper well-organized, comprehensive, and theoretically supported. Below, we have summarized your comments and provided our responses accordingly.

The details of how the gap of attention scores help the model to decrease the prediction probability on non-candidate labels are also not given.

Our paper utilizes margin-compliant loss to increase the gap between prediction probabilities for candidate and non-candidate labels, thus reducing the probability for non-candidate labels. To examine the impact of attention score gaps, we introduce MIPLMA-NOTEM, a variant of MIPLMA without the temperature parameter in the attention mechanism.

In experiments with the MNIST-MIPL dataset ($r=1$), both models yield similar prediction probabilities of approximately 0.088 for non-candidate labels, showing no significant difference. Visualization of attention scores for three multi-instance bags from the training set, shown in Figure R2 of the attached PDF, where the horizontal axis denotes the indices of instances, while the vertical axis represents the corresponding attention scores. Red and blue colors indicate attention scores assigned to positive and negative instances, respectively. The results reveals that the temperature parameter effectively increases the gap between attention scores for positive and negative instances.

Furthermore, Figure R3 of the attached PDF compares feature visualizations on the test set. MIPLMA produces well-clustered features by class, while MIPLMA-NOTEM exhibits errors, such as misclassifications where points from the first class incorrectly approach clusters of other classes. Therefore, while the temperature parameter may not directly impact prediction probabilities for non-candidate labels, it enhances feature representation accuracy. We will include these findings and discussions in the revised manuscript.

The margin-compliant loss seems not easy to be optimized in neural network framework. More details are required.

The core of the margin-compliant loss involves finding the maximum predicted probabilities for the candidate and non-candidate label sets, concatenating the margins, and calculating their mean and standard deviation. First, we use torch.max to obtain the maximum predicted probabilities for both sets. Then, we employ torch.cat to concatenate the margins in the current and previous batches. Finally, we calculate the mean and standard deviation of the margins using .mean and .std operations, followed by the value of the margin-compliant loss. All these operations, including torch.max, are differentiable in PyTorch. We will provide additional details on the margin-compliant loss in the revised manuscript to clarify its optimization within the neural network framework.

Will errors in the first few epochs affect subsequent training?

In MIPL, where prior knowledge of true labels in the candidate label sets is unavailable, initializing candidate label weights with averages is reasonable. Although early errors might accumulate and impact later training, our dynamic disambiguation loss reduces the influence of early predictions by assigning them lower weights, as shown in Eq. (8). To further investigate the effectiveness of our dynamic disambiguation loss, we also introduce a variant without the dynamic disambiguation, MIPLMA-$\alpha$, where $\alpha^{(t)}$ is set to 0 for all $t \in \\{1,2,\cdots,T\\}$.

The table below shows the performance of MIPLMA and MIPLMA-$\alpha$ on benchmark datasets. MIPLMA-$\alpha$ performs well on simpler disambiguation tasks (e.g., $r=1$ or $r=2$) but shows decreased performance compared to MIPLMA as the number of false positive labels increases (e.g., $r=3$). The gap between MIPLMA and MIPLMA-$\alpha$ widens with greater disambiguation challenges. These results suggest that our dynamic disambiguation loss is effective iin mitigating the errors in the first few epochs.

Thanks for your detailed and positive feedback on our paper. We are glad you found our approach to addressing ‘margin violations’ in the MIPL problem intriguing. Below, we address your comments and questions.

In Figure 1, a comparison with ELIMIPL and MIPLGP would be more beneficial.

MIPLGP follows the instance-space paradigm, which assigns augmented candidate label sets of bags to each instance and aggregates bag-level labels from instance-level labels. As a result, we cannot observe the phenomenon of margin violations in the instance and label spaces. For ELIMIPL, we will update Figure 1 to include its results in the revised manuscript.

The constant coefficient 0.95 in Equation 3 requires clarification.

The decay rate of $0.95$ for the temperature parameter in Equation 3 was selected based on preliminary experiments, which indicated that the performance of the margin-aware attention mechanism is relatively stable across a range of decay rates. To maintain consistency, we fixed the decay rate at $0.95$ for all datasets. We have conducted additional experiments on the FMNIST-MIPL dataset with decay rates from $0.9$ to $0.99$, as shown in Figure R1 of the attached PDF. These experiments revealed classification accuracies between $0.907$ and $0.927$. We will provide these details and results in the revised manuscript.

List detailed baseline performances on C-R34-16 and C-R34-25 datasets, as in Table 3.

Thank you for your suggestion. The following table shows the classification accuracies (mean±std) for MIPLMA, ELIMIPL, and DEMIPL on the C-R34-16 and C-R34-25 datasets. MIPLMA shows a slight improvement over ELIMIPL and DEMIPL on the C-R34-16 dataset, and a more significant advantage on the C-R34-25 dataset. We will include more baseline performances in Table 3 of the revised manuscript.

It remains unclear whether the superior performance of features learned by ResNet-34 holds on other benchmarks such as FMNIST-MIPL.

The instance-level features of the Birdsong-MIPL and SIVAL-MIPL datasets are preprocessed, which prevents the use of DCNNs like ResNet-34 for feature learning. We have tested LeNet and ResNet-34 on the MNIST-MIPL and FMNIST-MIPL datasets, but the classification accuracies were unsatisfactory. We believe this is due to the relatively simple nature of the features in these datasets, which are adequately captured by simple networks like the two-layer CNN used in our study. While deep convolutional neural networks might perform better on fully supervised MNIST and FMNIST datasets, their benefits may be less evident in weakly supervised scenarios.

Thank you for your detailed review and constructive feedback. In the following, we address your comments and concerns.

Definition of negative instances.

In line with previous MIPL work such as MIPLGP, DEMIPL, and ELIMIPL, we do not associate negative instances with the label space. For the MNIST-MIPL dataset, positive instances are drawn from the target classes $\\{0, 2, 4, 6, 8\\}$, while negative instances come from the reserved classes $\\{1, 3, 5, 7, 9\\}$. Positive instances in a multi-instance bag are sampled from target classes, and negative instances are sampled from reserved classes. Therefore, the true label of a multi-instance bag corresponds to the actual label, with candidate labels sampled from the remaining target classes.

Let us consider two scenarios:

1. Negative instances not associated with the labels space: If a test multi-instance bag has positive instances from class $0$ and negative instances from $\\{1, 3, 5, 7, 9\\}$, correct classification requires the classifier to predict class $0$.
2. Negative instances associated with the label space: If a test multi-instance bag has positive instances from $\\{0, 2\\}$ and negative instances from $\\{1, 3, 5, 7, 9\\}$, predicting either class $0$ or class $2$ is not considered wrong. However, this conflicts with the multi-class classification principle, which assumes each sample belongs to a single class.

Not associating negative instances with the label space is reasonable for MIPL applications. For example, For the CRC-MIPL dataset, positive instances are cell types like lymphocytes and colorectal adenocarcinoma epithelium, while negative instances refer to background information, such as non-cellular areas or areas without significant tissue. If background is included in the label space, the true label for each multi-instance bag should reflect both the background and the cell type class, conflicting with the multi-class classification principle that each sample should belong to a single class.

The margin-aware attention mechanism and margin-compliant loss are not novel.

We recognize that the attention mechanism and margin-based loss have been used in MIL and PLL, respectively. However, our paper introduces a novel perspective by identifying margin violations in both the instance and label spaces with a dual margin adjustment strategy. Existing margin-based PLL methods, such as PL-SVM and M3PL, have two main drawbacks: they rely on iterative optimization, making integration with neural networks difficult, and they only maximize the mean margin of predicted probabilities. In contrast, our margin-compliant loss integrates seamlessly with neural networks and both maximizes the mean margin and minimizes the standard deviation of predicted probabilities.

While the attention mechanism and the margin-based methods are not novel in MIL or PLL, their effective integration and enhancement within the MIPL framework represent a significant contribution. We will elaborate on these unique aspects and improvements in the revised manuscript.

Existing works decrease the novelty of this work.

Existing works, such as MIPLGP, DEMIPL, and ELIMIPL, provide important prior contributions to the MIPL framework. These can be divided into two categories: (1) MIPLGP uses a Gaussian Process Regression model for fitting transformed MIPL data, but it is sensitive to negative instances and requires more computational resources; (2) DEMIPL and ELIMIPL leverage attention mechanisms to learn global feature representations, yet they encounter margin violations in both instance and label spaces.

Our work distinguishes itself by being the first to identify and address margin violations in MIPL. We propose an effective dual margin adjustment strategy to counter these issues. Additionally, we introduce CRC-MIPL datasets with deep feature extractors, specifically the C-R34-16 and C-R34-25 datasets, which is also a novel contribution in MIPL.

The proposed algorithm does not achieve dominant performance on all the datasets under all the settings.

In machine learning, the "No Free Lunch" theorem states that no algorithm can outperform all others across every task and data distribution. Table A4 in the Appendix presents the results of pairwise t-tests at a significance level of 0.05 between our proposed MIPLMA and the comparison algorithms. Out of $430$ comparisons, MIPLMA outperforms the comparison algorithms in $412$ cases and shows no significant difference in $15$ cases. Thus, while MIPLMA may not dominate in every scenario, it demonstrates substantial superiority in the majority of cases.

Thank you for your insightful comments and positive feedback on our paper. We are glad you found the idea of margin regularization and our results valuable. Below, we answer your questions.

More discussion on setting the parameter $\lambda$. The parameter could significantly affect learning performance and might be better if varied during training rather than kept constant.

To address the concern about the parameter $\lambda$, we conducted further study using a dynamic adjustment strategy defined as $\lambda(t) = \min\\{\frac{t}{T^\prime} \lambda^\prime, \lambda^\prime\\}$, where $t$ denotes the current epoch, and $T^\prime$ and $\lambda^\prime$ control the rate of increase and the maximum value of $\lambda$, respectively. We name this model MIPLMA-$\lambda$. When $T^\prime=1$, MIPLMA-$\lambda$ is equivalent to MIPLMA. We conducted experiments on the MNIST-MIPL dataset with $T^\prime$ set to $\\{1,10,50,100\\}$, using the same $\lambda^\prime$ as in MIPLMA. Results show that dynamically adjusting $\lambda$ can sometimes improve classification accuracy compared to a constant $\lambda$; however, in other cases it may perform worse. Notably, when $r=3$, MIPLMA-$\lambda$ tends to enhance accuracy more significantly.

We sincerely appreciate your suggestion for dynamic adjustment of $\lambda$. Designing effective adjustment strategies is indeed a valuable research direction. We will explore more refined dynamic adjustment methods for $\lambda$ in the future.

Dear Reviewer zb3e,

Thank you for your insightful feedback, which has greatly enhanced our paper. We have carefully addressed your concerns, including conducting experiments on the dynamic adjustment strategy for the parameter $\lambda$. If you have any further questions or concerns, we would be pleased to discuss them.

Best regards,

The Authors

Thank you for your detailed review and valuable comments aimed at improving our paper. Below, we provide a summary of your comments along with our corresponding responses.

Concerns regarding generalization due to low class numbers or false positives.

We conducted additional experiments on the SIVAL-MIPL dataset with higher number of false-positive labels ($r \in \\{4,5,6,7,8,9\\}$). The results in the following table show that MIPLMA achieves superior performance in all cases, which demonstrate its robustness to various number of false positive labels.

Lack of comparison with recent PLL algorithms like PiCO.

We did not compare our method with PiCO for two main reasons:

• PiCO relies on data augmentation to learn diverse features of partially labeled images. However, the features of the Birdsong-MIPL, SIVAL-MIPL, and CRC-MIPL datasets are preprocessed tabular data. This makes it challenging to apply data augmentation-based PLL methods to these datasets.
• The instances in the MNIST-MIPL, FMNIST-MIPL, C-R34-16, and C-R34-25 datasets are raw images, which allows data augmentation-based PLL methods to learn features for each instance. However, these methods cannot aggregate a bag of instances into a unified feature representation as they lack an attention mechanism or similar aggregation method.

In our submission, we compared our method with five PLL methods published between 2020 and 2022. We have now included an additional comparison with a PLL method called POP (ICML 2023). As shown in Table R1 and R2 in the attached PDF, the main results indicate that while POP performs better than other PLL methods in most cases, it is inferior to our MIPLMA. We will include the above discussion and the results of POP in the revised manuscript.

A direct adaptation of margin-based losses.

We acknowledge that margin-based losses have been studied in existing methods in PLL such as PL-SVM and M3PL. However, these methods have two drawbacks that make them unsuitable for the MIPL problem: they rely on iterative optimization strategies, making them difficult to integrate with neural networks; furthermore, they only maximize the mean margin of the predicted probabilities. Our proposed margin-compliant loss integrates naturally easily with neural networks and simultaneously maximizes the mean margin while minimizing the variance of predicted probabilities.

To further illustrate this, we propose a variant of MIPLMA named MIPLML, which directly utilizes the margin loss $\mathcal{L}\_{\text{ml}}$ (Eq. 9). Specifically, the loss functions of MIPLMA and MIPLML are $\mathcal{L}=\mathcal{L}\_{\text{d}} + \lambda \mathcal{L}\_{\text{m}}$ and $\mathcal{L}=\mathcal{L}\_{\text{d}} + \lambda \mathcal{L}\_{\text{ml}}$, respectively. The results show MIPLMA's superiority, demonstrating that a direct adaptation of margin-based losses achieve worse performance.

Applications need to be more convincing.

In medical diagnostics, whole slide images (WSI) are high-resolution microscopy images of stained tissue slides. These images are extremely large, often reaching gigapixel size. Neural networks cannot learn an entire WSI due to their architecture and GPU limitations. Typically, WSIs are divided into smaller, lower-resolution tiles (multi-instances) to enable neural networks to learn meaningful features. In the future, we plan to collect higher-resolution WSI datasets to extend the MIPL applications.

The creation of the synthetic datasets is unclear.

For the MNIST-MIPL, FMNIST-MIPL, and Birdsong-MIPL datasets, we can control the distribution of true instances. The SIVAL-MIPL dataset, derived from the classical MIL dataset SIVAL, is created by adding false positive labels to each multi-instance bag. We calculate and report the proportions of positive instances in each bag, including their maximum, minimum, mean, and median values. This detailed distribution information will be included in the revised manuscript.

The comparison should be made using the same feature extractors.

For the MNIST-MIPL and FMNIST-MIPL datasets, we use a two-layer CNN and a fully connected layer to extract instance-level features, which are then aggregated into a unified representation using the attention mechanism. The feature extractors for MIPLMA, ELIMIPL, and DEMIPL are consistent. However, the PLL methods cannot directly handle multi-instance bags or individual instances in MIPL because they lack of an aggregation mechanism to yield bag-level features and the candidate label sets are unknown during training. Therefore, it is not feasible to apply the same feature extractors across MIPL and PLLmethods.

References:

[B] Xu et al. Progressive purification for instance-dependent partial label learning. ICML 2023.

Dear Reviewer H7jM,

Thank you for your thoughtful and detailed feedback, which has greatly improved our manuscript. We have carefully addressed your concerns by providing additional results on false positives in the SIVAL-MIPL dataset, comparing our method with the POP PLL method, and designing a variant of MIPLMA. We also included explanations about MIPL applications and the creation of synthetic datasets. Should any issues remain unresolved, we are happy to discuss them further.

Best regards,

The Authors

Thank you for your feedback and for updating your rating.

We acknowledge that various PLL methods, such as POP [B], PRODEN [C], and LWS [D], use different backbones like linear models, MLP, and ResNet depending on the dataset. For example, PRODEN and LWS mainly use linear and MLP models for MNIST and FMNIST datasets, while ResNet and convNet are employed for CIFAR-10 dataset. In our experiments, LeNet and ResNet-34 performed worse than our two-layer CNN on the MNIST-MIPL and FMNIST-MIPL datasets. This is likely due to the relatively simple nature of the features in these datasets, where deeper networks like ResNet may not provide significant advantages in weakly supervised scenarios. Additionally, the Birdsong-MIPL and SIVAL-MIPL datasets are preprocessed tabular data, which makes ResNet unsuitable for these datasets.

For the C-R34-16 and C-R34-25 datasets, we used ResNet-34 as the feature extractor for both MIPLMA and the compared PLL algorithms. The table below presents the results of these algorithms, where Mean and MaxMin represent the aggregation strategies used to transform instance-level features into bag-level features. The results indicate that MIPLMA outperforms the compared PLL algorithms, even with the same feature extractor. We will include more detailed experimental results in the revised manuscript. Additionally, we plan to collect higher-resolution WSI datasets and the use of pre-trained backbones, including large language models, for better feature learning.

References:

[B] Xu et al. Progressive purification for instance-dependent partial label learning. ICML 2023.

[C] Lv et al. Progressive identification of true labels for partial-label learning. ICML 2020.

[D] Wen et al. Leveraged weighted loss for partial label learning. ICML 2021.