Do Multiple Instance Learning Models Transfer?

We thank the reviewer for their detailed feedback on areas to improve rigor and insights from our results.

Q1. Benchmarks and statistical measures

---

We have added benchmarks with finetuned GigaPath (Q1 of BdAq), as well as using CHIEF pre-training on the PC dataset (Q1 of BdAq). We show std due to character limits and will show confidence intervals in the final paper. Please see Q3 of Rh9m for std of few-shot experiments.

Q2. Baseline comparisons

---

We now include 1) mean pooling for all slide-level encoding experiments (GigaPath and CHIEF in BdAq Q1) and 2) selectively transferring early layers (motivated by [1]) Below, we report how different transfer methods affect finetuning performance on a pre-trained ABMIL model (three-layer MLP with gated attention aggregation). Following TransFusion's approach, we investigate finetuning performance as progressively fewer layers are transferred. Starting with the attention module, we re-initialize the attention layer to random weights (Reset Attn), the third linear layer + attention (Reset Lin3+), the second and third linear layer + attention (Reset Lin2+), and all weights (Reset All).

We report performance relative to full weight transfer (PC108-L). Removing the attention layer causes the largest decrease (-5.0 average). Removing the remaining MLP layers results in a smaller but still substantial decrease (additional -3.3). These results highlight that the pretrained aggregation layer is crucial for successful supervised MIL pretraining. This contrasts with [1], which found that transfer of deeper layers has minimal effect on performance, emphasizing the unique nature of MIL transfer.

[1] Raghu, Maithra, et al. "Transfusion: Understanding transfer learning for medical imaging." Advances in neural information processing systems 32 (2019).

Q3. What features are most beneficial for transfer?

---

The experiment in Q2 suggests that the attention-based layer is the most beneficial for MIL transfer. To gain further insights, we next investigate the extent to which transferred layers change after finetuning. We follow the (SV)CCA approach used in [1], which measures the linear relationship between combinations of neuron activations across different models (0-100 scale). We compare activations before and after finetuning for each layer in standard (S) and large (L) ABMIL models on 30 randomly selected slides (45,232 patches) from the NSCLC-KRAS dataset.

Results below show pretrained layers change substantially less over the course of training than randomly-initialized ones. Most notably, the third and final attention layer (abmil c), exhibits extremely low correlation (2.5 & 16.3 for ABMIL-L and -S) with its original layer activations after training with random initialization, while pretrained models maintain high similarity (88.7 & 96.6). Since abmil c is the critical attention layer responsible for converting each patch embedding into a scalar weight for slide-level aggregation, this finding suggests that MIL transfer benefits heavily from transferring learned aggregation strategies.

Q4. What do you hypothesize is being captured?

---

Based on our results indicating the transferability of the attention layer, we hypothesize that pan-cancer pre-training provides a better starting point for the MIL model, by focusing on cancerous regions and disregarding regions of low diagnostic importance. The characteristics of tumor regions, such as pleomorphic nuclei, increased cellular density, and entropic cellular arrangement are common motifs across various tasks. Meanwhile, regions such as smooth muscle, processing artifacts, and red blood cells are consistently of minimal diagnostic relevance. Whereas training an MIL model from scratch leads to challenges learning to disregard these background patches, our results suggest that models trained on a large pan-cancer classification task are already equipped to prioritize tumor morphologies prior to finetuning.

We thank the reviewer for their valuable suggestions to further explore slide foundation models and clarify contributions. We provide our response below.

Q1. Retraining CHIEF

---

We train a new model (PC-CHIEF) on the PC dataset using CHIEF's training recipe, comprised of supervised contrastive loss and CLIP-based text embeddings of anatomical site.

Our evaluation shows that PC-CHIEF achieves a mean performance of 69.2 across tasks, which is slightly lower than the original CHIEF's 69.8. This similar performance suggests that dataset quality is not the primary factor explaining the performance gap between the PC-108 model and CHIEF. Notably, PC-CHIEF underperforms our PC-108 model by a large margin, despite being trained on the same data samples with additional textual embeddings and contrastive loss. This result highlights our pre-training approach as a simple but effective means of developing highly transferrable MIL models. We show performance averaged across tasks within each dataset, with number of tasks indicated in parentheses. All results use CTransPath features.

To further explore the efficacy of supervised pretraining, we extended our investigation to a different slide FM, Gigapath [1]. Unlike CHIEF, Gigapath was trained using a fully self-supervised approach, using a dataset of 171,189 WSIs. We compare finetuning performance of Gigapath with ABMIL pretrained on PC-108. All results use Gigapath patch features.

The ABMIL pretrained model, with a pretraining set only 2% the size of Gigapath, demonstrated substantially higher performance compared to the Gigapath slide FM. These results underscore the effectiveness of our pretraining approach even when trained with a substantially less data.

[1] Xu, Hanwen, et al. "A whole-slide foundation model for digital pathology from real-world data." Nature 630.8015 (2024): 181-188.

Q2. Limited contributions to methodological advancements.

---

While our work does not propose technical novelty in the form of a new MIL architecture, we respectfully disagree that our work makes limited contributions to methodological advancement. We note that in general ML, most high-impact papers on transfer learning are published using hypothesis-driven experimentation that emphasize unique scientific insights instead of methodology as a research contribution (Line 077-089). Not only is empirical investigation on transfer learning a valid and important research direction, but we also emphasize that research on MIL transfer is almost entirely absent in CPath. The vast majority of CPath studies in ML/CV conferences train MIL models from scratch, which may stem from lack of understanding on when and where these models would benefit from transfer. Furthermore, we provide an accessible, supervised alternative to slide foundation models trained on massive WSI cohorts (60-200k) via self-supervised learning, which demand substantial data and computing resources.

Q3. Limited Novel Insights

---

Though the benefits of transfer learning are well-established in the broader ML community, the absence of existing works on MIL transfer signals a distinct gap in the literature. As the first work to investigate MIL transfer, we reveal pan-cancer pretraining (considered by qGZD as a novel technique) as a powerful and overlooked approach to improve performance across all tasks and MIL methods, allowing a pre-trained ABML model to outperform SOTA slide foundation models (CHIEF and GigaPath) while requiring only 2-7% of the training samples. In addition, we also provide practical insights specific to MIL transfer, such as the importance of different layers for transfer (see response to 5tA4 Q2-3), the effect of model size on transfer (Rh9m Q1), the effect of task difficulty (Rh9m Q4), and the transferability of models across different organs (Figure A1).

Q4. Textual edits

---

We agree that inclusion of patch foundation models will provide a comprehensive overview of related works. We will include mention of models such as Virchow in the revision. We will adjust the typesetting and reference issues in the revision.

Thank you for your detailed feedback and for sharing your enthusiasm on the robust empirical evidence and strong performance of transferring MIL models. We have sought to address nearly all suggestions in additional experiments. Further details are provided below.

Q1. Pretraining strategies

---

To address the reviewer’s request, in addition to comparing against CHIEF, a SOTA slide foundation model (FM) trained on 60k WSIs (Table 2), we also:

1. Compared against Gigapath, another SOTA slide FM trained on 171k WSIs. We show similar findings that ABMIL pretrained on PC-108 (3,499 WSIs) outperforms Gigapath (see BdAq Q1).

2. Implemented CHIEF’s vision-language (VL) pretraining on PC-108 for fair comparison on pretraining strategy. Our results indicate conventional supervised pretraining outperforms VL pretraining on the same dataset (see BdAq Q1).

Q2. Limited originality

---

We respectfully disagree regarding the lack of substantial originality. Despite significant interest in MIL architectures for computational pathology (CPath) in ML/CV conferences, there is little to no investigation on MIL transfer in CPath. This is in stark contrast to progress made in general ML, with many high-impact papers using hypothesis-driven experimentation that contribute unique scientific insights (Line 077-089) without introducing new architectures [1,2]. Despite the importance of transfer learning in ML broadly, pretrained MIL model transfer remains unexplored in CPath due to limited understanding of its success conditions (Line 057-076, 102-109).

As the first work to investigate supervised MIL transferability in CPath, we believe that this research question is not only substantially original, but also impactful by highlighting a new avenue for obtaining highly generalizable slide-level representations. Despite progress made with self-supervised slide FMs, our alternative approach of supervised MIL model transfer has not been previously explored (Line 090-101). We show that our approach outperforms these methods while using less than 10% of the training data. Furthermore, we provide extensive insights into MIL transfer, such as how which features are transferred, how important each layer is to transfer, and how model size, patch encoders, MIL architecture, and task difficulty affect transfer.

[1] Kornblith, Simon, et al. "Do better imagenet models transfer better?" CVPR (2019): 2661-2671

[2] Fang, Alex et al. "Does progress on ImageNet transfer to real-world datasets? Neurips 36 (2024)

[3] Ericsson et al. "How well Do Self-Supervised Models Transfer?" CVPR (2021): 5414-5423

Q3. Visualization

---

We agree further interpretability is valuable. To this end, we also provided t-SNE visualizations showing pretrained slide embeddings differentiate classes better than random initialization (Figure 5). Since the scope of this work is on investigating MIL models, we preclude segmentation tasks, which focus on categorizing pixels rather than slides.

Motivated by the reviewer’s suggestion, we generated additional attention-based heatmaps for ABMIL on BRACS and NSCLC subtyping, finding that pretrained models focus on diagnostically relevant tumor regions even before finetuning. This indicates that the aggregation layer is highly transferable between tasks, which we further validated through quantitative explainability experiments (see 5tA4 Q3-4). Updated visuals and discussions will be included in our final submission.

Q4. Novel architectures

---

We have conducted extensive experimentation on 11 MIL architectures, with one of them also being graph-based (WikG). Though novel architectures are not the focus of our study, we modified a few MIL architectures to further explore the impact of model size on transferability (see Rh9m Q1). Results showed increased performance with larger model sizes compared to training from scratch, encouraging exploration of new high-capacity MIL models despite limited dataset size.

Q5. Cross-domain transfer

---

While investigating transfer from natural to pathology images is interesting, there is a scarcity of giga-pixel natural images to properly assess MIL model transfer. Instead, we had included transfer results using ResNet50 patch feature encoder (Table 3) trained on natural images, which showed significantly poorer performance than in-domain patch feature encoders.

Q6. Other comments

---

Novel Datasets: As our work evaluates 11 MIL models across 21 pre training tasks, contributions such as novel benchmark datasets could not be fully explored, though we also note that our study is potentially one of the largest in ML/CV conferences. Statistical tests: We provide standard deviation via 100 trials of bootstrapping for our updated results (see Bda4 Q1, 5tA4 Q3, Rh9m Q1, Q3) and will provide confidence intervals in the main text. Suggested reference: We clarify that "How well do self-supervised models transfer?" was cited in our submission (Line 082-083).

We thank the reviewer for their extensive suggestions on further improving the rigor and insights from our work.

Q1. Model Size

---

We add larger ABMIL models (7M and 9M parameters) and Transformers at comparable sizes, finding:

1. ABMIL performance plateaus in the 5-7M range, with a notable decrease at 9M parameters.
2. At smaller sizes (0.2M and 1M), ABMIL and Transformer achieve similar performance both with and without pretraining.
3. At larger sizes, Transformers benefit substantially from pretraining, even outperforming ABMIL at 7M. Similar to ABMIL, Transformers show reduced performance at 9M.

We display the performance averaged across BRCA ER/PR, NSCLC TP53/STK11, GBM C/F, BRACS C/F.

Q2. Why does ABMIL perform best?

---

ABMIL is consistently effective across CPath tasks and patch encoders [1]. In the data-restricted regimes common in CPath, the default transformer configuration may be overparameterized. Our results show that transformers can achieve comparable performance to ABMIL at smaller parameter counts and through pretraining at larger model sizes.

[1] Campanella, Gabriele, et al. "A clinical benchmark of public self-supervised pathology foundation models." arXiv (2024).

Q3. Few-shot

---

The target tasks in Figure 3 included molecular (NSCLC TP53/STK11/EGFR, BCNB ER/PR/HER2, GBMLGG C) and morphological classification (BRACS C/F). We included BRACS C/F, as PC-108 contains only invasive carcinoma cases (a single label in BRACS F). We recognize this could raise concerns and will exclude BRACS in the revision.

Shown below is performance averaged over ABMIL, TransMIL, Transformer, DFTD, and CLAM for molecular tasks alone and BRACS alone, confirming that few-shot learning benefits are robust to tasks without any label overlap.

Q4. Task Difficulty

---

To compare transfer quality between easy and hard tasks, we perform a paired t-test on transfer performance between the easiest and hardest task for each dataset. Given a fixed pretraining dataset, this allows us to investigate how the difficulty of the training objective affects the transferability of the final model. For morphological datasets, we assign coarse and fine subtyping as easy and hard tasks. For molecular datasets, which typically have multiple tasks (e.g BCNB ER/PR/HER2), we compare the model performance (averaged over ABMIL, DFTD, TransMIL, Transformer trained from scratch) on each task, selecting the task with the lowest and highest AUC as the hard and easy task, respectively. This resulted in the following:

For each pair of pretraining tasks, we compare finetuning performance across the 17 remaining evaluation tasks for four MIL models, resulting in a total of 476 paired points (4 models x 7 pretraining x 17 evaluation). The result shows that pretraining on hard tasks leads to better transfer performance, with average improvement of +0.5 (95% CI=0.1-0.9, p=0.017). This indicates that challenging training objectives enhance the transferability of the final model.

Q5. PC Transferability

---

Although ABMIL shows more variability in KNN performance, Figure A2 shows that PC pretraining consistently achieves strong average performance across all models. Specifically, PC-108 ranks second-highest in ABMIL and highest in both TransMIL and Transformer KNN evaluations. Figures 2 and A2 both therefore support our conclusion that PC pretraining produces robustly transferable features.

Our finetuning results in Figure A1 further validate this finding, showing that although there is some variability, PC pretraining consistently outperforms random initialization and achieves the highest average performance among pretraining tasks across all models, including ABMIL. Thus, we believe our conclusion from Figure 2 is well validated by our finetuning results. We will make this connection clearer in the final submission.

Q6. Suggestions

---

Multitask: We agree that multi-task pretraining is a promising next step and will investigate this in further iterations. Data Splits: We share splits between tasks from the same dataset to ensure fair comparisons. Patch encoders: We will add Virchow 2G results in the final paper. Data availability: To facilitate further work on this topic, we will release the model weights and code for PC108/43 pretrained models.