Revisiting Subsampling and Mixup for WSI Classification: A Slot-Attention-Based Approach

• Weakness 1

Thank you for providing a detailed comparison.

We would like to highlight that our method achieves an AUC of 0.975 for CAMELYON-16 and an AUC of 0.981 for TCGA-NSCLC, showcasing better-calibrated predictions with improved Negative Log-Likelihood (NLL) values. These results are obtained using features extracted from SimCLR, provided by the DSMIL paper[4].

As previously outlined in our general response, the overall performance disparities observed in various papers can be attributed to the challenges posed by deficient and unbalanced datasets. The significance of a standardized evaluation protocol becomes crucial in such scenarios. Notably, TRANSMIL stands out as the most parameterized model in the baseline, making it particularly vulnerable to issues like overfitting and changes in the validation set. In the case of CAMELYON-16, TRANSMIL[3] exhibits varying AUC values of 0.877[1], 0.906[2], and 0.931[3].

For further details on our efforts to ensure fair and reproducible comparisons, we invite you to refer to our general response.

• Weakness 2

In terms of the subsampling rate (p), the empirical evidence from Table 6 and Table 8 in the Appendix shows that adopting any subsampling rate results in a performance gain compared to not adopting it. This implies that the benefits of subsampling can be harnessed without needing a prior understanding of the ratio of positive patches. While we reported the optimal subsampling rate(p), it's worth noting that improvement from the baseline model can still be achieved without precisely tuning the subsampling rate(p).

• Weakness 3

The main hyperparameters in our method comprise the number of slots (S), mixup beta distribution alpha (α), subsampling rate (p), and Late-mix (L), as elaborated in Section 4.3 of the paper. We fix the optimal hyperparameters based on the results of k-fold AUC. AUC is recognized as a more robust metric than ACC, as it is not influenced by threshold variations.

For Slot-MIL, where S is the sole hyperparameter, we observe minimal performance differences when the number of slots exceeds a certain threshold. Especially in TCGA-NSCLC, there is no discernible trend when varying the number of slots. It's important to highlight that our approach is based on learnable (implicit) clustering using attention scores, making meticulous hyperparameter search unnecessary, unlike non-learnable k-means clustering. Additional experiment results are provided in the table below.

In the case of Slot-MIL + SubMix, we employ grid search to select hyperparameters, with α ∈ {0.2, 0.5, 1.0}, p ∈ {0.1, 0.2, 0.4}, and L ∈ {0.1, 0.2, 0.3}, while maintaining the same number of slots as in Slot-MIL. Similar to subsampling rate (p) and the number of slots (S), the results show minimal changes with variations in α and L. It's important to note not to use an excessively large α (beyond 1.0) or to avoid using Late-mix. Detailed results can be found in Table 9 in our paper.

• Weakness 1

Thanks for your constructive review.

However, we want to emphasize that our method is a novel approach and contributes a lot to MIL problems, while the underlying ideas may exist in other domains. You mentioned DeepAttnMISL, but we consider DeepAttnMISL to be notably different from ours for two main reasons: clustering mechanism and pooling strategy.

Firstly, in the clustering phase, while seemingly achieving similar goals, DeepAttnMISL utilizes non-learnable k-means clustering, which cannot be trained with backpropagation or end-to-end. However, our approach employs a learnable O(n) complexity attention-based (implicit) clustering mechanism, which has already been explored in many different deep learning tasks including sets [7], object-centric learning [8], etc. Also, our method is much simpler to implement, not needing a manual action to capture clusters. This crucial distinction allows our method to adapt more dynamically to the underlying data structure, enhancing its efficacy.

Moreover, in the pooling phase, DeepAttnMISL relies on local attention, whereas our model integrates an O(n) complexity global attention pooling strategy. This strategic choice contributes to the model's ability to capture dependencies between clusters, which may be particularly valuable in tasks involving complex relationships. Again, our method is simple, computationally efficient, robust to distribution shifts, and able to capture underlying structures well, leading to state-of-the-art performance.

It's worth noting that non-learnable k-means clustering has its limitations, as some previous studies have indicated; its adaptation results in a marginal improvement of performance [1], and its application is confined solely to intra-label mixup[2]. Also, k-mean clustering shows optimal performance only in the limited range of the number of clusters (k) (as shown in Table 3 of [3], and Fig.A.1 of [2]), while our Slot-MIL shows the consistent performance when the number of slots (S) exceed a certain threshold.

Furthermore, our novel attention clustering, with its linear time complexity, extends its utility to inter-label mixup, thereby addressing a broader spectrum of scenarios. To the best of our knowledge, the mixup between WSIs using a fixed number of attention-based clusters remains unexplored by other researchers.

• Weakness 2

Thanks for your insightful comparison.

We forgot to mention the results with SimCLR features in the main paper, which was already detailed in Appendix B.6. With comparable contrastive-learning-based features, our model achieves a notable performance of 0.923/0.975 (ACC/AUC), demonstrating superior results compared to ILRA-MIL and other papers discussed in the context of C16. We'd like to underscore that the performance is significantly influenced by the type of pre-trained features and evaluation protocol considering the scarcity and imbalance in the overall train/validation/test sets. We encourage you to refer to our general response at the top where we elaborate on our dedicated efforts to ensure fair and reproducible comparisons.

• Weakness 3,4

We appreciate your comments, and we find that applying our method to gene mutation prediction or multi-class Gleason grading in PANDA is very intriguing. Recognizing the potential limitations of C16 and TCGA-NSCLC for evaluating sophisticated MIL approaches, we intentionally included the CAMELYON-17 (C17) experiment in Table 4.

C17, being a multi-center dataset with a substantial distribution shift, offers a more realistic reflection of real-world scenarios compared to other datasets. In the case of C17, both Slot-MIL and Slot-MIL + SubMix outperform the baselines by a large margin with well-calibrated predictions. It's worth noting that C17 has been recognized as challenging, even with pre-training[5] or surgical fine-tuning[6] has shown limited promise. Notably, our method addresses distribution shifts in C17 without the need for pre-training or fine-tuning.

Bayes-MIL[4] also reveals its performance on C17 in their appendix Table 5. However, they didn't specify whether they partitioned the multi-center data to make distribution shifts which is common practice on C17. It's noteworthy that our method achieves superior performance on C17 without relying on a slide-dependent Regularizer (SPDR) or incorporating spatial information (APCRF), both of which are essential components in Bayes-MIL.

Your understanding and consideration are greatly appreciated. By reflecting your feedback, we are planning to reorganize the updated version of the paper to highlight the results of C17, emphasizing the strong performance gap. If you have any further questions or require additional clarification, please don't hesitate to reach out.

Best regards, Authors

Below are additional questions and concerns following the rebuttal:

Do we need better MIL if improving SSL encoder demonstrates the most performance gains?: The reviewer thanks the authors for updating their results (Table 5) with self-supervised features using SimCLR. However, these results suggest a different finding – that developing more principled MIL architectures or augmentation strategies is not as significant as using a better SSL encoder. In CAMELYON-16 results in Table 5, ABMIL reaches the same AUC as base Slot-MIL (with smaller standard deviation) and with lower NLL (better supposed calibration). Slot-MIL is matched in AUC by DTFD-MIL. Potentially, ABMIL with subsampling can do better than Slot-MIL with SubMix (baselines presented in Table 5 do not seem to have subsampling). In TCGA-NSCLC results in Table 5, all MIL methods reach the same AUC as Slot-MIL (with or without Submix). The reviewer also acknowledges the author’s rebuttal in that “not all tasks possess an optimal feature extractor”. However, it is becoming increasingly prevalent with histopathology domain-specific encoders such as CTransPath [1] that SSL is the mainstay approach for extracting features in MIL. As suggested, evaluating on more difficult tasks may highlight better performance gains with Slot-MIL. However, the current results significantly diminishes Slot-MIL’s contributions.

Confusion with calibration: NLL is used as the main approach for measuring calibration. Upon review, the reviewer is unsure why NLL is used, when metrics such as Brier score, conformal prediction, and Expected Calibration Error (ECE) are some of the main methods for evaluating calibration [2,3].

Missing comparison with CLAM-SB: Upon review, the results of this work do not compare against CLAM-SB [4,5,6], which has been demonstrated to be a strong baseline (commonly evaluated in other MIL works on these tasks). As the evaluated tasks can be solved at the instance-level, comparing against CLAM-SB is necessary.

Concluding Thoughts The reviewer thanks the authors for the detailed reply. From reviewing the submission again, I did find the experimentation and investigation of subsampling as an augmentation strategy to be interesting and thoughtful. This is a worthwhile investigation, and I appreciate both positive and negative results that would make salient how subsampling works. However, the evaluation on NSCLC subtyping and CAMELYON-16, coupled with the updated results via SSL, may not be challenging enough to demonstrate the novelty of Slot-MIL and SubMix. The evaluation of SubMix itself is not convincing, as established calibration metrics are not used. The performance gains are extremely reduced following using SSL features, which emphasizes further interesting points on: 1) trade-off between MIL architecture and patch-level encoder, 2) what new clinical problems can MIL architectures solve that can't be solved with a strong patch-level encoder? At the moment, I am not changing my rating, but would encourage the authors to continue working on this problem as I believe it would have important results if evaluated on more diverse tasks.

1. Wang, X., Yang, S., Zhang, J., Wang, M., Zhang, J., Yang, W., Huang, J. and Han, X., 2022. Transformer-based unsupervised contrastive learning for histopathological image classification. Medical image analysis, 81, p.102559.
2. Naeini, M.P., Cooper, G. and Hauskrecht, M., 2015, February. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence (Vol. 29, No. 1).
3. Minderer, M., Djolonga, J., Romijnders, R., Hubis, F., Zhai, X., Houlsby, N., Tran, D. and Lucic, M., 2021. Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems, 34, pp.15682-15694.
4. Lu, M.Y., Williamson, D.F., Chen, T.Y., Chen, R.J., Barbieri, M. and Mahmood, F., 2021. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering, 5(6), pp.555-570.
5. Xiang, J. and Zhang, J., 2022, September. Exploring low-rank property in multiple instance learning for whole slide image classification. In The Eleventh International Conference on Learning Representations.
6. Yufei, C., Liu, Z., Liu, X., Liu, X., Wang, C., Kuo, T.W., Xue, C.J. and Chan, A.B., 2022, September. Bayes-MIL: A New Probabilistic Perspective on Attention-based Multiple Instance Learning for Whole Slide Images. In The Eleventh International Conference on Learning Representations.

The reviewer thanks the authors for their thoughtful response. After reading the authors’ rebuttal, thoughts of other reviewers, and inspecting the methodological details further (of this work and its preceding works), Presented below is a point-by-point reply:

Reply to Weakness 1 - Novelty of Slot-MIL: Slot-MIL is indeed distinct from DeepAttnMISL (not requiring clustering beforehand) for addressing patch redundancy. At the same time, as the methodology of Slot-MIL is more of a direct extension of the already-existing Slot Attention paper, it would be nice to visualize and characterize which slots are associated with clinically relevant pathological features to further justify adapting this method from other literature. Any number of methods can be intuitively adapted from Transformer literature and broader works in deep learning literature for addressing redundancy (e.g. - I imagine Clustering Transformers would also work well here [1]). As the performance of Slot-MIL is ultimately modest, there does not seem to be a strong motivation to use Slot-MIL when ABMIL, DeepAttnMISL, TransMIL, and IRLA-MIL (combined with self-supervised features) already work very well. In addition to visualization of slots, other ways to demonstrate the applicability of Slot-MIL is with additional experimentation regarding: 1) solving harder tasks and 2) data-efficiency experiments. The improvement with SubMix (as raised by vwP9 and its reply) is only in NLL, which does not translate into any improvement of clinically relevant performance metrics.

Reply to Weakness 2 – On “SOTA” performance: The reviewer understands that experimental designs (splits, parameters, implementation) differ across studies, especially when comparing MIL results with different pretrained encoders for feature extraction. The intention of my critique was to lessen the tone, as the previous (and also current versions) of this work continue to emphasize “SOTA” performance.

Reply to Weakness 3 – Diverse Benchmarks: The reviewer understands that C17 is a good benchmark for studying distributional shift. At the same time, the reviewer contends that the experiments in this study are too limited to binary classification tasks. As utilized in many works, PANDA presents a more challenging task with 1) multi-class classification with label uncertainty, and 2) domain shift. The suggestion of PANDA (and other tasks) is two-fold in improving the study:

1. Finding tasks where Slot-MIL would have better improvement: As the results stand, there is limited performance gain for Slot-MIL (and its combination with SubMix) (see below).

2. Task diversity: Even with C17, the only tasks evaluated are lung cancer subtyping and breast cancer metastasis detection. These tasks are diagnostically simple and over-represented in computational pathology studies, relative to much harder tasks such as gene mutation prediction and even Gleason score grading.

3. Vyas, A., Katharopoulos, A. and Fleuret, F., 2020. Fast transformers with clustered attention. Advances in Neural Information Processing Systems, 33, pp.21665-21674.

The baselines listed above the double line in Table 5 do not incorporate augmentation (and also subsampling), and the baselines below the double line use augmentation. To be clear,

• DTFD-MIL employs a pseudo-bag with knowledge distillation based on the ABMIL model but does not exhibit significant improvement. Here, a pseudo-bag splits a single WSI into subset patches, which can be seen as a variant of subsampling.
• RankMix, built on our Slot-MIL along with subsampling and patch-level mixup, displays inferior performance compared to SubMix, underscoring the importance of mixing at the attention-based clustered feature level. The rationale behind conducting RankMix on Slot-MIL lies in Slot-MIL being identified as the most powerful model. RankMix authors emphasized the significance of baseline performance in achieving optimal results.
• Also, as demonstrated in Table 6 (Appendix B.1), even when subsampling is applied to ABMIL and DSMIL, it does not align closely with the performance achieved by SubMix, demonstrating a substantial margin.
• We also conducted experiments with other baselines (TRANSMIL, ILRA) incorporating subsampling and found that they do not closely match SubMix's performance by a considerable margin. Additionally, applying subsampling to TRANSMIL is impractical due to its limited applicability, to be strict. The model relies on the ordering of full patches for the utilization of an additional positional encoding CNN module.

We can provide a further analysis within a day if needed.

In summary, our experiments indicate that even when subsampling is applied to the baseline, leading to performance improvement through the regularization of attention over-concentration on specific patches, it does not closely align with the performance achieved by SubMix.

• Weakness 1

Thanks for your comprehensive comparison.

We want to highlight that our enhancements amount to approximately 9.4%, 6.5%, and 3.1% when measured against the best baselines in terms of AUC for CAMELYON-16, CAMELYON-17, and TCGA-NSCLC, respectively. While this improvement might seem modest, it's crucial to note that we have concurrently improved calibration while enhancing both ACC and AUC, making this margin noteworthy.

As detailed in Table 3, we acknowledge that Slot-Mix doesn't yield a substantial improvement in terms of ACC and AUC. However, its contribution to calibration is significant, especially as it involves mixing inter-label slides using informative slots. Notably, our mixup approach stands out for its simplicity and efficiency, especially when compared to other augmentations such as DTFD-MIL[1] and Rank-Mix[2], which necessitate knowledge distillation based on grad-cam[3] or a pre-training phase to unify the number of patches between slides. It's also important to highlight that DFTD-MIL and Rank-Mix naturally pose similarities to subsampling, involving the division of WSIs into pseudo-bags and selecting a subset of patches. The comparison with these baselines becomes more fair in the context of Slot-MIL + SubMix. In response to reviewer Wtm8, we also incorporate additional metrics about the complexity at reply for Wtm8.

• Weakness 3, Question 1

The ratio of positive patches in a positive slide varies across datasets, as it is known to be less than 10% in CAMELYON-16 and over 80% in TCGA-NSCLC[4]. The precise number of patches containing relevant signals for classification is naturally unknown. While subsampling may potentially discard some important patches in certain iterations, the stochasticity introduced by subsampling and the abundance of patches within a slide mitigate the likelihood of discarding all useful patches. This can be further adjusted by the subsampling rate (p).

Retaining all patches, as depicted in our "Base train" in Figure 3, exacerbates overfitting since attention scores become excessively concentrated on specific patches. Additional details are provided in Section 4.1.2. Regarding the subsampling rate (p), empirical evidence in Table 6 and Table 8 in the appendix demonstrates that adopting any subsampling rate leads to performance gain compared to not adopting it. This implies that the benefits of subsampling can be harnessed without requiring a prior understanding of the ratio of positive patches.

We already validate the effectiveness of subsampling in CAMELYON-16, where only a small proportion of patches may contain relevant signals for classification. Also, at inference stage, we utilize whole patches in a slide, not a subset patches of a slide. The optimal range of subsampling rates might be higher in scenarios where the ratio of positive patches is low, as low subsampling rates may result in the exclusion of any positive patches (p=0.1 indicates the use of 10% of total patches per iteration)

• Weakness 2, Question 2

The adoption of subsampling for multi-class scenarios is anticipated to pose no significant obstacle, as subsampling is an orthogonal augmentation with the number of classes, akin to dropout rates[5] and masking ratios in MAE[6]. While determining an optimal subsampling ratio (p) might require further elaboration in multi-class scenarios, our observations suggest that the performance is not overly sensitive to p, making this task more manageable.

It is worth noting that Slot-Mix originated from manifold mixup[7], where multi-class scenarios are the basic premise. Consequently, the adoption of Slot-MIL + SubMix in multi-class situations is not only feasible but also aligns with the method's conceptual foundations.

Demonstrating the validity of our method on multi-class datasets, such as PANDA, where WSIs are categorized into 6 classes based on Gleason score, would indeed be valuable. However, given the large scale of PANDA with tens of thousands of WSIs, feature extraction might require a substantial amount of time over the rebuttal period. If you are aware of any open-source repositories providing extracted features from PANDA, we are keen to conduct experiments as time permits. Regarding the code, it has already been posted in the supplementary materials for Slot-MIL. We are in the process of making it open-source and endeavor to release it anonymously within the coming week including SubMix.

If you have any further questions or need additional clarification, please feel free to reach out.

Sincerely, Authors

For 3), to our best knowledge, using mixup augmentation with a fixed number of intermediate features is a very new approach in MIL problems, while being simple to implement and cost-efficient than other SOTA augmentation methods[2,6]. In addition, our experiment on CAMELYON-17 with a huge distribution shift, which is not experienced much in the WSI classification area as they are challenging[7], empirically proved that our method is much more robust than other methods. As real-world application inevitably suffers from distribution shift, our performance is quite notable. Moreover, we want to underscore the importance of our well-calibrated predictions, a crucial factor in the medical domain where reliability and explainability are needed.

Lasty, we will address each reviewer's concerns as soon as possible. We appreciate your valuable feedback and encourage any additional questions or requests for further experiments. Please feel free to stay in touch with us.

Best regards, Authors

Experiment with SimCLR-based features (Also can be found in Appendix B.6). We use the identical hyper-parameters and evaluation protocol as the experiment with ResNet-extracted features. | Method\Dataset | | CAMELYON_16 | | | TCGA-NSCLC | | |:-----------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:| | | ACC(⭡) | AUC(⭡) | NLL(⭣) | ACC(⭡) | AUC(⭡) | NLL(⭣) | | Meanpool | 0.693±0.000 | 0.604±0.003 | 0.674±0.023 | 0.927±0.014 | 0.972±0.010 | 0.232±0.055 | | Maxpool | 0.920±0.002 | 0.967±0.002 | 0.353±0.147 | 0.920±0.023 | 0.961±0.013 | 0.608±0.053 | | ABMIL | 0.921±0.009 | 0.972±0.002 | 0.234±0.033 | 0.933±0.018 | 0.981±0.010 | 0.263±0.101 | | DSMIL | 0.916±0.012 | 0.968±0.008 | 0.456±0.182 | 0.936±0.017 | 0.981±0.010 | 0.324±0.133 | | TRANSMIL | 0.889±0.026 | 0.939±0.010 | 0.988±0.188 | 0.924±0.020 | 0.974±0.009 | 0.381±0.127 | | ILRA | 0.923±0.009 | 0.973±0.007 | 0.333±0.043 | 0.933±0.018 | 0.981±0.011 | 0.277±0.103 | | Slot-MIL | 0.922±0.008 | 0.972±0.007 | 0.294±0.065 | 0.937±0.018 | 0.981±0.011 | 0.276±0.131 | | Slot-MIL + SubMix | 0.923±0.009 | 0.975±0.008 | 0.229±0.071 | 0.935±0.018 | 0.981±0.012 | 0.248±0.105 |

References

[1]Xiang, Jinxi, and Jun Zhang. "Exploring low-rank property in multiple instance learning for whole slide image classification." The Eleventh International Conference on Learning Representations. 2022.

[2]Zhang, Hongrun, et al. "DTFD-MIL: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[3]Shao, Zhuchen, et al. "Transmil: Transformer based correlated multiple instance learning for whole slide image classification." Advances in neural information processing systems 34 (2021): 2136-2147.

[4]Li, Bin, Yin Li, and Kevin W. Eliceiri. "Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[5]Ilse, Maximilian, Jakub Tomczak, and Max Welling. "Attention-based deep multiple instance learning." International conference on machine learning. PMLR, 2018.

[6]Chen, Yuan-Chih, and Chun-Shien Lu. "RankMix: Data Augmentation for Weakly Supervised Learning of Classifying Whole Slide Images With Diverse Sizes and Imbalanced Categories." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[7]Yufei, Cui, et al. "Bayes-MIL: A New Probabilistic Perspective on Attention-based Multiple Instance Learning for Whole Slide Images." The Eleventh International Conference on Learning Representations. 2022.