Revisiting End-to-End Learning with Slide-level Supervision in Computational Pathology

Thanks for providing feedback and taking the time to review our work! We have provided point-by-point clarifications on the issues and supplemented the experiments as suggested.

W1&W2: Clarification on E2E and Foundation Model

R1:

• First, we would like to clarify that E2E and Pathology Foundation Models (FM) are not opposing concepts. We acknowledge that FMs have made extensive applications and prominent contributions in pathology due to their effectiveness and generalization. However, we argue that the performance of traditional patch-based FMs is approaching saturation [1], and an increasing number of FMs are adopting E2E-based self-supervised pre-training [2, 3]. We believe that E2E with slide-level supervision primarily advantages in its training process being more aligned with clinical tasks (rather than merely learning from medical images themselves) and its more efficient data utilization. A concurrent work holds a similar view [4]. We have shown that small ResNet models pre-trained only on ImageNet-1k can achieve performance comparable to or even superior to that of two-stage schemes based on computationally expensive pre-trained FMs, with E2E training costing <10 3090 GPU-hours (a cost similar to, or even lower than, the training cost of some latest large-scale slide encoders [3]).

• More importantly, we supplement the following external validation to demonstrate the generalization ability of encoders trained via E2E:

  • TCGA->CPTAC	R50+ABMIL	R50+TransMIL	R50+WIKG	UNI+ABMIL	UNI+TransMIL	UNI+WIKG	R50+ABMILX (E2E)
    CPTAC-NSCLC (AUC)	66.42	74.59	64.04	83.73	85.24	83.56	85.19
    CPTAC-LUAD (C-index)	46.34	48.24	OOM	53.59	51.36	OOM	54.00

OOM = Out-of-Memory on 24GB-RTX-3090

Without any fine-tuning, the model trained on TCGA was directly tested on CPTAC-sourced data. The R50 trained via E2E not only showed significantly improved generalization but even outperformed UNI.

• Furthermore, constrained by resources, this work is limited to small-scale models and datasets. However, we believe that E2E, after addressing optimization challenges, has greater potential in pathology. We conducted a preliminary attempt at UNI by applying ABMILX for E2E Parameter-Efficient Fine-Tuning (PEFT) on UNI. The following results demonstrate the advantages of ABMILX in E2E tasks and preliminarily verify the potential of E2E to further enhance FMs. We note that work [5] adopted a similar approach but overlooked the impact of optimization challenges caused by MIL on E2E training.

  • 	UNI	R50 (E2E)	UNI+PEFT (E2E)	UNI+PEFT (E2E)	UNI+PEFT (E2E)
    	ABMIL	ABMILX	ABMIL	TransMIL	ABMILX
    PANDA	74.69	78.83	74.11	76.61	80.99

• 【Patch Sampling】 In fact, we only perform patch sampling during the training phase, with sampling varying across iterations to ensure diversity. This approach is relatively common not only in general domains [6] but also in Pathology [7]. We have conducted comprehensive ablation studies on the sampling strategy in both the main text and supplementary materials, and found that it not only achieves optimal efficiency but also helps eliminate data redundancy and enhance data diversity.

W3&Q3: Clarification on Latest Baselines and Two-stage Details

R2: In fact, we compared the performance of WIKG and RRT under FM features (UNI, CHIEF, GIGAP) in Table 1 of the Supplemental Material. In the table below, we have reorganized these results and supplemented 2DMamba [8]. In the revised version, this content will be included in the main text. Furthermore, except for the training of the GIGAP aggregator (due to memory constraints), all other two-stage experiments did not employ any random sampling strategy. Additional details are provided in Supplemental Material B.3.

OOM = Out-of-Memory on 24GB-RTX-3090

Q1&Q2: ABMIL vs. ABMILX on more applications

R3: Multiple fair ablation studies comparing ABMILX and ABMIL are presented in both the main text (Figure 1(c), Sec.3.3, Sec.4.3) and supplemental material (Table 2, Table 3, Sec.D). Here, we reorganize and summarize their comparisons in E2E and Two-Stage settings, and supplement the comparisons in more scenarios (Pavement Distress Classification, Diabetic Retinopathy Grading, Path GCN framework).

• 【Clarification of details】 The data in Figure 1(a) are identical to those in Table 1. E2E+ResNet in Figure 1(a) corresponds to E2E+ResNet-50+ABMILX in Table 1. The possible reason for the minor discrepancy is that Table 1 only includes one challenging benchmark, while Figure 1 is more comprehensive.

• 【E2E & Two-Stage】 We have summarized the table below from the main text and supplemental material. All comparisons in the table ensure that all other settings are identical except for the MIL model.

  • 	E2E	E2E	E2E	Two-Stage	Two-Stage	Two-Stage
    	Grading	Subtyping	Survival	Grading	Subtyping	Survival
    ABMIL	75.46	89.23	62.70	71.85	94.39	63.81
    ABMILX	78.34	93.97	67.78	73.01	94.83	66.34

• 【More Tasks】 Following [9], we supplemented experiments on E2E recognition of high-resolution images, such as Pavement Distress Classification (CQU-BPDD [10]) and Diabetic Retinopathy Grading [11], to more thoroughly compare ABMIL and ABMILX.

  • Task\Model	ABMIL	TransMIL	ABMILX
    CQU-BPDD	81.3	82.0	82.5
    DRG	75.3	74.3	78.1

• 【PathGCN】 We supplemented comparisons with PathGCN, demonstrating the effectiveness of ABMILX when applied to other frameworks.

  • PathGCN+\Task	Grading	Subtyping	Survival
    ABMIL	76.90	93.64	64.97
    ABMILX	77.92	94.23	65.96

[1] PathBench: A comprehensive comparison benchmark for pathology foundation models towards precision oncology.

[2] A Multimodal Knowledge-enhanced Whole-slide Pathology Foundation Model.

[3] A whole-slide foundation model for digital pathology from real-world data. Nature.

[4] Foundation Models - A Panacea for Artificial Intelligence in Pathology?

[5] Unlocking adaptive digital pathology through dynamic feature learning.

[6] Masked Autoencoders Are Scalable Vision Learners. CVPR 2022.

[7] Multiple instance learning framework with masked hard instance mining for whole slide image classification. ICCV 2023.

[8] 2DMamba: Efficient State Space Model for Image Representation with Applications on Giga-Pixel Whole Slide Image Classification. CVPR 2025.

[9] No Pains, More Gains: Recycling Sub-Salient Patches for Efficient High-Resolution Image Recognition. CVPR 2025.

[10] An iteratively optimized patch label inference network for automatic pavement distress detection. T-ITS.

[11] Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening. Information Sciences.

Thank you for your appreciation and suggestions. However, we would still like to make some clarifications regarding the framework issue and the scalability of E2E learning. First of all, what we would like to clarify is that our method is not limited to random sampling or small-scale encoders. Constrained by resources, we adopted a combination of small-scale encoders and random sampling. However, its actual performance is surprising, which is highly consistent with the viewpoint of [4]. This work also achieved performance close to that of foundation model (FM) encoders by end-to-end (E2E) training small-scale encoders with random sampling, challenging the current high-cost foundation model pretraining. Regarding works [2, 3] cited in our rebuttal, we would like to clarify that although both serve as frozen feature encoders in downstream tasks, [2] indeed utilized random sampling and slide aggregators in its pretraining phase to optimize the patch encoder. We suggest this can be regarded as an application of E2E in pretraining. As for sampling strategies, we have actually discussed them sufficiently in the manuscript. We consider the choice of sampling strategy as a trade-off between performance and efficiency. In the reorganized table below, we investigated the impact of different sampling strategies on E2E training and found that while different sampling strategies have little impact on performance, they significantly affect training efficiency. Moreover, random sampling does not actually reduce performance significantly. Our proposed MRIS, after supplementing multi-scale information to alleviate the problem of the model's local vision that may be caused by random sampling, achieves even better performance.

Furthermore, if further reduction of E2E memory consumption is required (e.g., when training foundation models), we have also validated the use of recently proposed dual-buffer sampling—wherein a subset of patches from those used in forward propagation is selected for backward propagation. ABMILX and the E2E framework can still achieve substantial performance while maintaining extremely low memory consumption.

On the selection of different encoders: to demonstrate the scalability of our framework and method, we have replaced the small-scale encoder with the foundation model UNI in this rebuttal. This is a preliminary attempt, where we applied MRIS and ABMILX for PEFT on UNI. Results in the table below preliminarily indicate that the advantages of MRIS and ABMILX on small-scale models can be extended to foundation models. ABMILX still achieves significant performance gains by mitigating optimization challenges, while MRIS also exhibits stable performance (The computational cost would be prohibitive if more complex sampling strategies were adopted in end-to-end training with foundation-level encoders, whereas MRIS only required ~40 3090 GPU hours).

Additionally, we are following the reviewers' suggestions to explore the performance of E2E trained encoders compared to FM under MIL methods such as PathGCN. However, we would like to emphasize that the experiments in our supplementary material—where ABMILX is applied to two-stage FM features—demonstrate that its performance is highly competitive. We consider it to be an advanced MIL method.

When the performance of the two-stage paradigm is approaching its bottleneck, E2E in computational pathology remains a vast uncharted territory. Its true prosperity, popularity, and generalization will inevitably require the participation of researchers across the entire community in the future. As stated in our title, this work aims to revisit E2E while exploring its potential and challenges to the best of our ability, and we suggest we have made every effort to achieve this. We would like to once again express our gratitude for your patience, appreciation, and suggestions.

Thank you again for the helpful discussion. As the deadline is approaching, we just wanted to quickly and respectfully check if our most recent response has resolved your remaining concerns. Please let us know if anything remains unclear.

W1: Novelty and contribution

R1:

【Innovation in research direction】We would like to clarify that the main contribution of this paper lies in Revisiting slide-level E2E work in computational pathology and proposing that the overlooked sparse MIL models in E2E are actually the primary bottleneck hindering the performance improvement. We define this as the E2E optimization challenge, with its mathematical definition provided in the Supplemental Material. This differs entirely from the focus of previous E2E papers, which mainly concentrated on improving sampling strategies and reducing memory usage, yet still showed significant performance gaps compared to foundation model-based two-stage schemes.

【Architectural novelty】Based on this E2E optimization challenge, we propose a concise yet effective MIL model, ABMILX. Therefore, the design motivation of this model differs from the common attention-based aggregators mentioned by the reviewer, which focus on the two-stage paradigm with offline feature inputs. Additionally, the design of the model is novel, and Reviewer igAe has expressed strong interest in the ABMILX structure. Specifically, we introduce a multi-head mechanism to the classic sparse architecture ABMIL to alleviate the extreme sparsity of the original architecture. More importantly, we utilize global feature correlation to refine local attention scores. This utilization of global correlation is pioneering—existing methods typically utilize global correlations to refine instance features rather than instance attention scores. Furthermore, we suggest that compared to existing works, which often compute local attention and global attention independently, our idea of using global attention to refine the local one is also a quit ecreative attention combination manner. For instance, as shown in our experimental results, RRT-MIL, which uses global attention and local attention independently, achieves significantly worse performance than our ABMILX in E2E learning. It is precisely the optimization challenge caused by sparse attention scores that motivated our innovation. We provide rigorous and comprehensive proofs in the Supplemental Material regarding how these two mechanisms reduce optimization risk.

【Experimental and community contributions】We have conducted a comprehensive and systematic evaluation of E2E and the proposed ABMILX, demonstrating the potential of E2E in computational pathology. Comparisons were performed across 7 datasets, 4 different tasks, 3 state-of-the-art foundation models of varying sizes, and 9 different MIL aggregators. Without any pre-training on pathological data, with a training cost of only 10 3090 GPUHours, our method outperforms foundation models on multiple challenging benchmarks while maintaining a significant inference efficiency advantage. Furthermore, we supplemented results showing its generalization comparable to foundation models, as well as the excellent performance of ABMILX on non-pathological datasets or other application settings. More importantly, to demonstrate the potential of E2E, we supplemented experiments on ABMILX and the proposed E2E framework in UNI PEFT. Due to space constraints, details are provided in responses R1 and R3 to Reviewer igAe.

W2&Q1: Ablation of ABMILX and MRIS

R2: In fact, we have already provided comprehensive and detailed ablation experiments in our paper (Sec.3.3, 4.3, and Table 3 in main text) regarding the performance contributions of ABMILX and MRIS. Here, we have reorganized and summarized a clearer version, as shown in the table below. Moreover, it should be clarified that we did not claim in the original paper that MRIS could address optimization risks. It is merely a sampling strategy that efficiently incorporates multi-scale information while keeping low time cost. And ABMILX is designed to mitigate the optimization risks in the E2E process.

W2: The comparison with sampling strategies

R3: In fact, our paper has already presented sufficient sampling strategies for comparison. These include naive random sampling (RS) and our MRIS, as well as cluster sampling (C2C), naive attention sampling (Attention), and variational information bottleneck-based attention sampling (FT) (which fall under selective sampling strategies). Here, we have reorganized and summarized a clearer version, as shown in the table below. We suggest these experiments validate one of our insights: in the E2E process, the role of MIL in optimization is often more pronounced than that of sampling strategies.

Q2: Latest Baselines

R4: In fact, we compared the performance of WIKG and RRT under FM features (UNI, CHIEF, GIGAP) in Table 1 of the Supplemental Material. In the table below, we have reorganized these results and supplemented 2DMamba [1]. In the revised version, this content will be included in the main text.

OOM = Out-of-Memory on 24GB-RTX-3090

Q3: optimization collapse

R5: First, as stated in Section 3.3 of the main text, we have provided additional mathematical analysis on optimization collapse in the supplemental material (Sec.A), which will be incorporated into the main text in revised revisions. A concise mathematical explanation is that during the E2E process, the gradients received by the encoder $\theta$ for $i$-th instances consists of the gradients from the instance embedding $e_i$ and the gradients from the corresponding normalized attention score $\hat{a}_i$:

$\theta \leftarrow \frac{\partial \mathcal{L}}{\partial e_i}+\frac{\partial \mathcal{L}}{\partial \hat{a}_i}=\frac{\partial \mathcal{Z}}{\partial e_i}\cdot\frac{\partial \mathcal{L}}{\partial Z}+\frac{\partial \mathcal{L}}{\partial \hat{a}_i}= \frac{\partial \mathcal{\sum\hat{a}_j e_j}}{\partial e_i}\cdot\frac{\partial \mathcal{L}}{\partial Z}+\frac{\partial \mathcal{L}}{\partial \hat{a}_i}=\hat{a}_i\frac{\partial \mathcal{L}}{\partial Z}+\frac{\partial \mathcal{L}}{\partial \hat{a}_i}$,

where $\mathcal{L}$ denotes the loss value and $Z=\sum\hat{a}_je_j$ denotes the aggregation of all instance embeddings. It could be concluded that the gradients from $i$-th instance embedding $\frac{\partial \mathcal{L}}{\partial e_i}=\hat{a}_i\frac{\partial \mathcal{L}}{\partial Z}$ are scaled by corresponding normalized attention score $\hat{a}_i$. And sparse attention tends to assign high scores to noisy instances $\mathcal{N}$, leading to optimization collapse. Therefore, we suggest using the maximum attention score of noisy instances as a metric to quantify optimization collapse $\mathcal{R}$:

$\mathcal{R} = O(\max_{i \in \mathcal{N}} \hat{a}_i).$

To better demonstrate optimization collapse, we use the product of the maximum attention score of noisy instances for each slide and the total number of instances as the final metric (MAX-N). Furthermore, we have measured MAX-N on the CAMELYON dataset in the early training stage. The results are reported in the table below, demonstrating that ABMILX effectively mitigates optimization collapse while maintaining reasonable sparsity. These experiments and analyses will also be added to the revised version. Finally, in page 6 of the main text, we have demonstrated how ABMILX mitigates optimization collapse through visualization.

Sparsity is a metric proposed in the main text to measure model sparsity, and it can also be regarded as an indirect metric of optimization collapse.

[1] 2DMamba: Efficient State Space Model for Image Representation with Applications on Giga-Pixel Whole Slide Image Classification. CVPR 2025.

See title. Please acknowledge that you have read through things and respond to the authors.

Dear Reviewer tziM,

As the discussion phase draws to a close, we are writing to make one final check-in. Your feedback has been critical in shaping our revisions, and in our rebuttal, we’ve worked diligently to address each point you raised. We sincerely look forward to your corresponding conscientious feedback of high-quality to help us polish our work while helping the Area Chair assess the work with full context. We understand your schedule is demanding and deeply value the time you’ve already invested. And If our revisions have met your expectations, we would appreciate your consideration in updating your evaluation.

Thank you again for your valuable time.

Dear Reviewer tziM,

With the discussion phase ending soon, we wish to ensure we have adequately addressed all your valuable concerns. Your initial feedback was instrumental in improving our work, and we are eager to know if our responses and the updated manuscript meet your expectations. Please let us know if you have any remaining questions. Should our response resolve your primary concerns, we would be grateful if you would consider updating your evaluation.

Thank you for your valuable time.

Thanks for providing feedback and taking the time to review our work! We have provided point-by-point clarifications on the issues and supplemented the experiments as suggested.

W1: Clarification on Theoretical Justification

R1: As stated in Section 3.3 of the main text, we have provided a comprehensive and rigorous mathematical analysis of ABMILX in the Supplemental Material (Sec.A). This includes the definition of optimization risk, as well as complete proofs demonstrating how the proposed Multi-head Local Attention Mechanism and Attention Plus Propagation reduce optimization risk. It will be included in the main text in the revised version.

W2&Q2: The results of more medical tasks and clarification on real-world data

R2:

• 【Clarification on dataset used】 First, we would like to clarify that computational pathology is closely linked to cancer analysis, as the analysis of pathological images is typically regarded as the gold standard for cancer diagnosis and prognosis in clinical practice. Therefore, cancer datasets are commonly used in the field of computational pathology. Due to ethical concerns and the inaccessibility of private data, we utilized public datasets. However, the public datasets we used are diverse in sources (TCGA, CAMLYON, PANDA), covering various cancers across multiple organs (prostate, breast, lung, bladder) with rich and challenging task types (cancer subtyping, cancer grading, cancer patient survival time analysis, cancer metastasis detection). These datasets are derived from real-world clinical scenarios. For instance, TCGA, the largest cancer database in the United States, is not a benchmark specifically designed for model evaluation. Furthermore, these datasets vary in size (PANDA images typically have dozen of millions of pixels, while TCGA images usually exceed 1 billion pixels) and noise ratios (CAMLYON has less than 10% cancer cell area, whereas other datasets generally have over 40%). In addition, we supplemented external validation experiments using CPTAC data (another large cancer data source) to demonstrate the performance of our method across more clinical cases. The response to R3 provides specific results and details.

• 【Diabetic retinopathy grading】 Furthermore, we further verify the generalizability of our method to other high-resolution E2E medical tasks through diabetic retinopathy grading (DR) [1] and existing frameworks [2]. The results are reported in the table below, which demonstrate the advantages of the proposed ABMILX in optimizing high-resolution end-to-end tasks.

  • 	ABMIL	TransMIL	ABMILX
    DR (Acc)	75.3	74.3	78.1

W3: The generalization compared foundation models

R3: We have supplemented the following external validation on CPTAC to demonstrate the generalization ability of the encoder trained on TCGA via E2E learning:

OOM = Out-of-Memory on 24GB-RTX-3090

The experimental results demonstrate that the R50 optimized via an E2E approach on TCGA not only exhibits significant improvements in generalization but even outperforms UNI (a ViT-L model pre-trained on 1B-scale pathological data). This validates the strengths of both E2E learning and our proposed ABMILX in terms of generalization ability and transferability across real-world variations.

Q1: Clarification on hyperparameters

R4: We analyzed the impacts of different global attention scaling factors and numbers of attention heads in Section 3.3 of the main text and Section C.4 of the Supplemental Material, respectively. We have reorganized these experiments in the table below:

We use a learnable global attention scaling factor by default, while the number of heads has an impact on model performance. However, selecting 8 or 4 by default can achieve sufficiently strong performance without the need for additional hyperparameter tuning.

Q3: Gaps between the Theoretical Analysis and the Experimental Results

R5: According to our theoretical analysis in the Supplemental Material, the maximum attention score of noisy instances can serve as a representative of optimization risks. And our theoretical analysis suggests that ABMILX can mitigate this risk while maintaining reasonable sparsity. Therefore, we use the product of the maximum attention score of noisy instances for each slide and the total number of instances (MAX-N) to validate the gap between theoretical analysis and the experimental results**.** And we have measured MAX-N on the CAMELYON dataset in the early E2E training stage. The results are reported in the table below. As consistent with the theoretical analysis, ABMILX indeed mitigates optimization risks effectively while maintaining reasonable sparsity. This proves that there is no significant gap between our theoretical analysis and experimental results. These experiments and analyses will also be added to our revised version. A minor gap is that our experimental results have shown that the value of $\alpha$ has a certain impact on the final performance, but this was not fully discussed in our theoretical analysis. We suggest that the influence of the value of $\alpha$ stems from its role in controlling the attention sparsity of MIL. In the future, we will conduct corresponding theoretical analyses and experiments to design a MIL architecture with better performance in terms of sparsity.

[1] Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening. Information Sciences.

[2] No Pains, More Gains: Recycling Sub-Salient Patches for Efficient High-Resolution Image Recognition. CVPR 2025.

Dear Reviewer AkkT,

With the discussion period drawing to a close, we would like to follow up and gently check if our rebuttal has addressed your concerns to your satisfaction. We are on standby to provide any further clarifications you might need. If you are satisfied with our responses, we would be very grateful if you would consider updating your evaluation.

Thank you again for your time and valuable feedback. We look forward to hearing from you.