Cancer Survival Analysis via Zero-shot Tumor Microenvironment Segmentation on Low-resolution Whole Slide Pathology Images

Thanks for your constructive comment. Here are the responses to Weaknesses (W) and Questions (Q).

W1. Definition of zero-shot learning.

We apologize for the misunderstanding caused. In fact, our method follows a transductive zero-shot learning approach. As a matter of fact, the generalized zero-shot learning can be divided into two settings: inductive and transductive. In the inductive setting, only information from seen classes is used during training. In contrast, the transductive setting allows access to partial information (e.g., class names or visual features) from unseen classes except for the ground truth labels in the training process. We will clarify this distinction in the revised version. Thank you for your valuable suggestion.

W2. Declaration of LLM usage.

We apologize for the inconsistency between our response in the NeurIPS checklist and the main body of the paper. We recognize that the use of GPT-4 to generate detailed text descriptions for different tissue categories is crucial in our method. We will update the checklist in the revised version to accurately reflect the role of LLMs in our approach. Thank you for pointing this out.

W3. There is no experiment designed to verify if ZTSurv actually learns the pattern.

We thank the reviewer for such insightful comments. Due to the limitation of uploading the visualization results in the rebuttal phase, we instead follow the study in [1], which calculates the mean edge weights between tumor and lymphocyte nodes in the constructed graph for long-survival and short-survival patients based on the patient stratification results in Fig.2. Intuitively, the weights of the edges that connect tumor and lymphocyte should be higher for patients in low survival risk group, since the strong interactions between lymphocyte and tumor regions will indicate a brisk immune response to human cancers [2], and the experimental results denoted below table is also consistent with the above immune knowledge that can demonstrate the effectiveness of our method.

[1] Zuo, Yingli, et al. "Identify consistent imaging genomic biomarkers for characterizing the survival-associated interactions between tumor-infiltrating lymphocytes and tumors." International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2022.

[2] Saltz, Joel, et al. "Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images." Cell reports 23.1 (2018): 181-193.

Q1. How would the performance be affected if the initial TIL map is of low quality or contains noise.

Thanks for this great comment. To assess the robustness of our model to label noise in TIL maps, we randomly perturb 5% and 10% of the labels in the original TIL map and evaluate their performance for both segmentation and survival prediction tasks.

Segmentation results under different label noise ratio:

Survival prediction results under different label noise ratio:

As can be seen from the tables above, although the prediction results for both segmentation and survival prediction tasks are slightly decreased when the label noises are introduced, our method is general robust to the TIL maps with low-level label noise, showing its potential in real-world clinical applications, where perfectly labeled data is often limited.

Once again, we sincerely thank you for your thoughtful review and constructive feedback. If you have any further questions, we would be glad to respond. If you find our responses satisfactory, we would greatly appreciate your reconsideration of the score.

We sincerely appreciate the time and effort you have devoted to reviewing our submission. We have carefully prepared our rebuttal, aiming to address the valuable points you raised. Should you have an opportunity to review our response, we would be grateful for any additional thoughts or clarifications you may wish to share. Thank you once again for your thoughtful and constructive feedback.

Best regards,

The Authors

Thanks for your constructive comment. Here are the responses to Weaknesses (W) and Questions (Q).

W1. ZTSurv transfers annotation burden upstream.

Previous methods require training a segmentation or classification model that includes all target classes, necessitating manual annotations for each of these categories. In contrast, our approach only requires annotation of the TIL map, significantly reducing the annotation burden.

W2 & W5. Zero-shot segmentation lacks ground-truth validation of tumor and stroma.

We thank the reviewer for such great comment. We would like to clarify that currently, there is no complete semantic segmentation dataset available for TCGA, and the only existing annotation is the TIL map. As a result, quantitative evaluation is limited to the seen class (lymphocyte), for which we report mIoU. Since ground truth annotations for unseen classes (tumor and stroma) are unavailable, direct evaluation of these classes is not feasible. To address this limitation and assess the reliability of our segmentation results, we leverage the foundation model PLIP to compute the image–text alignment between segmented image patches and class-specific text prompts. Specifically, we evaluate the similarity between: (1) tumor patches and the text prompt “tumor”, (2) tumor patches and the text prompt “stroma”, (3) stroma patches and the text prompt“tumor”, and (4) stroma patches and the text prompt“stroma”. We compute the average scores for each of these pairs across four datasets. The results are presented in the table below.

From the results, we observe that the similarity between image and text embeddings of the same tissue type is consistently higher than that between different tissue types, which to some extent reflects the accuracy of our segmentation for unseen classes (tumor, stroma).

W3. ZTSurv's tissue ablation study in Table 3 excludes necrosis.

We thank the reviewer for such great comment. We conduct an additional experiment incorporating necrosis, and here are the results:

From the experimental results, it is evident that including necrosis indeed improves survival prediction, which aligns with the findings in [1]. We acknowledge this limitation in the Appendix H.7, where we note that including a broader range of TME components, such as blood vessels, necrosis, and fibroblasts, could further enhance predictive performance. In future work, we will explore these additional tissue types to capture a more comprehensive representation of the TME and improve the robustness of our model.

[1] Wu, Junxian, et al. "Leveraging tumor heterogeneity: Heterogeneous graph representation learning for cancer survival prediction in whole slide images." Advances in Neural Information Processing Systems 37 (2024): 64312-64337.

W4. Graph construction based on KNN only relies on feature similarity and ignores spatial distance.

We apologize for the confusion. To clarify, our graph construction method based on KNN incorporates spatial distance (L2 norm). We will correct this in the revised version. Furthermore, to address this concern more directly, we conduct an additional experiment where the graph is constructed based on feature similarity. Here are the results:

Q1. ZTSurv does not leverage GPT-4-generated class descriptions for node representation.

We appreciate this valuable suggestion. We conduct a comparison experiment where we incorporate GPT-4-generated class descriptions for node representation. Here are the results:

From the results, incorporating GPT-4-generated class descriptions for node representation contributes to improved performance on multiple datasets. We again appreciate your insightful suggestion.

Q2. Survival loss backpropagation to segmentation lacks verification. & Q3. How are the weight coefficients in the loss function determined.

We thank the reviewer for raising these important questions. Recent studies have demonstrated that accurate modeling of the TME is critical for reliable survival prediction [1] [2] [3]. This suggests that segmentation and survival prediction are inherently aligned objectives in the context of histopathology analysis. To verify the effect of survival-guided segmentation, we compare the performance of zero-shot segmentation with and without survival guidance in Table 2 of the original manuscript. The results indicate that incorporating survival supervision leads to improved segmentation accuracy, supporting the effectiveness of this joint optimization strategy.

However, we agree with the reviewer that the balance between segmentation and survival loss is crucial. In our formulation, this trade-off is controlled by the hyperparameter $\gamma$. To examine its effect, we conduct experiments under different $\gamma$ values and report both survival prediction and segmentation performance across four datasets.

Segmentation results under different $\gamma$ values:

Survival prediction results under different $\gamma$ values:

From these results, we observe that when $\gamma$ is set to a very small value or to 0, as in the original Table 2 where no survival guidance is applied, segmentation lacks prognostic supervision and tends to capture less clinically meaningful structures, leading to reduced segmentation accuracy. In contrast, when $\gamma$ is too large (for example, $\gamma$=50), the survival loss dominates the training objective. This can mislead the segmentation module, as it begins to overfit to survival-related global signals at the expense of precise spatial delineation.

For the other coefficients in the loss function i.e., $\alpha$ and $\beta$ are set following the configurations from previous work [4].

[1] Wu, Junxian, et al. "Leveraging tumor heterogeneity: Heterogeneous graph representation learning for cancer survival prediction in whole slide images." Advances in Neural Information Processing Systems 37 (2024): 64312-64337.

[2] Shao, Wei, et al. "Tumor micro-environment interactions guided graph learning for survival analysis of human cancers from whole-slide pathological images." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Han, Minghao, et al. "Multi-scale heterogeneity-aware hypergraph representation for histopathology whole slide images." 2024 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2024.

[4] Zhou, Ziqin, et al. "Zegclip: Towards adapting clip for zero-shot semantic segmentation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.

Once again, we sincerely thank you for your thoughtful review and constructive feedback. If you have any further questions, we would be glad to respond. If you find our responses satisfactory, we would greatly appreciate your reconsideration of the score.

We sincerely appreciate the time and effort you have devoted to reviewing our submission. We have carefully prepared our rebuttal, aiming to address the valuable points you raised. Should you have an opportunity to review our response, we would be grateful for any additional thoughts or clarifications you may wish to share. Thank you once again for your thoughtful and constructive feedback.

Best regards,

The Authors

Thanks for your constructive comment. Here are the responses to Weaknesses (W) and Questions (Q).

W1&Q1. How does the proposed method compare to recent slide-level foundation models.

Thanks for this great comment. We agree that recent slide-level foundation models such as TITAN, PRISM, CHIEF are promising advances toward modeling spatial and contextual information across WSIs. However, they typically adopt a two-stage design, where the modeling of TME structures and survival prediction are handled independently that miss the inherent correlation among them. Moreover, they rely heavily on high-resolution patch-based preprocessing, which requires considerable computation and memory for tiling and patch-level feature storage. In contrast, our method simultaneously realize the TME segmentation and survival prediction tasks in an end-to-end manner on the down-sampled WSIs, which enables joint modeling of spatial tissue structures and prognostic outcomes while reducing computational cost.

We conduct a comparison using the WSI-level embeddings generated by TITAN, PRISM, and CHIEF for survival prediction. As shown below, our approach outperforms these methods in survival prediction across four cancer types. Moreover, according to your suggestions, we will introduce these slide-level foundation models in the related work of the revised manuscript.

W2&Q2. How does the resolution affect the segmentation and survival prediction performance.

We fully agree that spatial resolution can influence performance, particularly for tasks that rely on fine-grained histological detail. However, our method is specifically designed to capture the distributional patterns of TME components such as TILs at the whole-slide level, rather than relying on detailed cellular morphology. As demonstrated in [1], the spatial patterns of TILs are closely associated with patients’ survival, highlighting the power of spatial distribution analysis in addressing clinically relevant questions, even without high-resolution morphological detail.

Moreover, as discussed in the response to W3 of reviewer UaL5, we chose 50x downsampling because the utilized TIL maps are labeled under such resolution [1], and thus we are unable to test the performance of our method on the TIL maps with higher resolution. By contrast, according to your suggestion, we conduct additional experiments to examine the impact of lower-resolution inputs by further downsampling the TIL map to 100×. The results are shown below, along with the average inference time per WSI implemented on a single NVIDIA RTX 4090 GPU.

Survival prediction results:

Segmentation results:

As shown in above tables, lowering the resolution to 100× leads to a consistent decline in performance across all cohorts. This suggests that excessive downsampling reduces the amount of useful information available for both tissue segmentation and survival prediction.

[1] Saltz, Joel, et al. "Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images." Cell Reports 23.1 (2018): 181-193.e7.

W3. Potential label noise introduced at tissue boundaries or mixed-type regions.

We thank the reviewer for such insightful comment. Actually, the original WSIs are typically scanned at a spatial resolution of 0.25 μm/pixel. After downsampling the images by a factor of 50, the resulting spatial resolution becomes: 0.25 μm/pixel × 50 = 12.5 μm/pixel. This resolution aligns well with the average size of a single cell, which is typically around 10–15 μm in diameter.

At this scale, each pixel approximately corresponds to the size of a cell with specific type, and thus the scenario where each pixel in the down-sampled WSIs corresponds to a different tissue type is unlikely happen. By contrast, if the downsampling level achieves to 100x (i.e., 25.0 μm/pixel), each pixel in the down-sampled image will have more chance to involve multiple cells with diverse types that will lead to ambiguous or misleading node representations. We believe this is the one of the reasons that the survival prediction results under the downsampling ratio of 50x is significantly better to that under 100x. Please see the result tables shown in the answer to your previous comment (W2 &Q2).

Q3. Dominant tissue class within fixed-size regions.

We thank the reviewer for such great comment. Similar to previous works [1][2], we assign each graph patch a dominant tissue class label based on the most frequent tissue type within the patch, which has been shown effective in prior studies and our methods. We agree with you that such approximation maybe ineffective on tissues with high heterogeneity. A possible solution is to treat the patch with reduced size as node, but it will bring more graph nodes that will lead to higher computational complexity for graph learning. We will add it as the limitation of this study in the revised manuscript.

[1] Wu, Junxian, et al. "Leveraging tumor heterogeneity: Heterogeneous graph representation learning for cancer survival prediction in whole slide images." Advances in Neural Information Processing Systems 37 (2024): 64312-64337.

[2] Shao, Wei, et al. "Tumor micro-environment interactions guided graph learning for survival analysis of human cancers from whole-slide pathological images." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

W4&Q4. Similarity measures for graph edge construction.

We appreciate the reviewer’s thoughtful comments. To investigate the concern that Pearson correlation over concatenated visual and text embeddings may create spurious edges, we conduct a controlled similarity analysis across different types of node pair. Specifically, we compute the mean similarity scores for three types of relationships: (1) Image–Image (Same Type): Pairwise similarity between nodes of the same tissue class based on visual features only. (2) Image–Text (Same Type): Similarity between a node's image embedding and its corresponding text embedding (e.g., tumor image vs. “tumor” text). (3) Image–Text (Different Type): Similarity between a node's image embedding and text embeddings from different tissue classes (e.g., tumor image vs. “stroma” text). The results are summarized below:

From the results, we can observe that image and text embeddings of specific tissues are biologically related but not excessively redundant , which suggests that these two modalities provide complementary information. Consequently, computing Pearson correlation over their concatenated representations offers a meaningful and interpretable way to model interactions within the tumor microenvironment.

We also agree with the reviewer that a fixed similarity metric such as Pearson correlation might not fully capture complex or non-linear relationships. To explore this further, we implement a learnable similarity function inspired by attention mechanisms. Specifically, we project node embeddings into query and key spaces using learnable matrices and compute edge weights by taking the dot product between queries and keys, followed by a ReLU activation. Here are the results:

As shown in the table, the learned similarity approach achieves improved performance to fixed Pearson correlation. As part of our future work, we plan to conduct more detailed analyses and further explore the potential of data-driven approaches in this context.

Q5. How the baseline models were trained.

All baseline models were trained on full-resolution WSIs, following their original implementations. These methods rely on patch-level inputs extracted at high magnification and cannot be directly applied to down-sampled whole-slide representations. As a result, preprocessing steps such as tiling and patch-level feature extraction are required to prepare the data for these methods.

To further evaluate performance under a consistent input setting, we additionally compare our method with ProtoSurv [1] on 50× down-sampled WSIs. The results are shown below:

These results demonstrate that our method maintains strong performance at significantly lower resolution, highlighting its efficiency and scalability.

[1] Wu, Junxian, et al. "Leveraging tumor heterogeneity: Heterogeneous graph representation learning for cancer survival prediction in whole slide images." Advances in Neural Information Processing Systems 37 (2024): 64312-64337.

Once again, we sincerely thank you for your thoughtful review and constructive feedback. If you have any further questions, we would be glad to respond. If you find our responses satisfactory, we would greatly appreciate your reconsideration of the score.

Thanks for your constructive comment. Here are the responses to Weaknesses (W).

W1. The construction relies heavily on PLIP.

We thank the reviewer for such great comment. In fact, while your mentioned study [1] dose not use the PLIP embeddings, it relies on another foundation model i.e., UNI for patch-level feature extraction. For fair comparison between our method and the study in [1] , we apply the same encoder of VLM to extract features (i.e., PLIP and UNI embeddings), and the final survival prediction results are listed below:

As shown in the table above, for both PLIP and UNI encoders, our ZTSurv consistently outperforms [1] across all datasets (BRCA, UCEC, LUAD, BLCA, and Overall), which demonstrates the effectiveness of our approach for survival prediction of human cancers from WSIs.

[1] Wu, Junxian, et al. "Leveraging tumor heterogeneity: Heterogeneous graph representation learning for cancer survival prediction in whole slide images." Advances in Neural Information Processing Systems 37 (2024): 64312-64337.

W2. The technical contributions appear to be relatively incremental:

We appreciate your feedback and would like to to clarify several key points regarding our contributions at first, and then will address your comment point by point. The main contribution of this study is that we propose a novel end-to-end framework ZTSurv for survival prediction of human cancers that can segment different TME components and capture their spatial interactions for clinical outcome prediction. To the best of our knowledge, this is the first unified learning framework that can simultaneously conduct zero-shot tissue segmentation and survival prediction, which allows us to achieve better segmentation accuracy with low annotation burden and improve survival prediction performance, as these two tasks are highly related and thus can mutually enhance each other when learned together. More importantly, we achieve this on 50x down-sampled WSIs, which can significantly reduce the computational cost and storage burden in comparison with the existing studies that are implemented on high resolution whole-slide pathological images (WSIs).

• The use of VLM embeddings for zero-shot class prediction is common.

  While it is true that the VLM embeddings for zero-shot prediction have been explored, they mainly focus on classification or object detection tasks [2] [3], and it is still challenging to transfer the zero-shot learning ability of VLM to the tissue segmentation tasks on histopathological images and fewer studies work on it. Accordingly, we present a novel prototype guided zero-shot tissue segmentation model (Section 3.1 ) by the aid of patch-level PLIP embeddings that can achieve satisfied tissue segmentation results.

• Building a heterogeneous graph for representing TME components has been done.

  We acknowledge that heterogeneous graphs have been used to represent TME components for survival prediction in previous works, such as [1]. However, these methods follow a two-stage learning process where the TME patches should be firstly picked out from the whole-slide pathological images for the heterogeneous graph construction via training a tissue segmentation or patch-level classification model. Then, they rely on the constructed heterogeneous graph to build the clinical outcome prediction model. By contrast, we design an one stage learning paradigm that can simultaneously conduct the heterogeneous graph construction and survival prediction tasks in an end-to-end manner. The experimental results shown in Table 1 of the original manuscript clearly demonstrate the advantage of our method in comparison with the existing studies that treat the tasks of heterogeneous graph building and survival prediction independently.

• The edge creation is novel, but minor.

  Although the idea of concatenating the semantic information to the node visual embedding for edge representation is simple, the experimental results shown in Table 4 of the appendix clear show its advantage. Such observations also inspire to design more complicated ways to combine the semantic prompts with the visual representation of nodes for further performance improvement. We will point it out as the future research direction in the conclusion section of the revised manuscript.

[1] Wu, Junxian, et al. "Leveraging tumor heterogeneity: Heterogeneous graph representation learning for cancer survival prediction in whole slide images." Advances in Neural Information Processing Systems 37 (2024): 64312-64337.

[2] Guo, Miaotian, et al. "Multiple prompt fusion for zero-shot lesion detection using vision-language models." International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2023.

[3] Kim, Sanghyun, et al. "Locality-Aware Zero-Shot Human-Object Interaction Detection." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

W3. The choice of 50x downsampling appears to be somewhat arbitrary.

Thanks for this great comment. We chose 50x downsampling because the utilized TIL maps are labeled under such resolution [1], and thus we are unable to test the performance of our method on the TIL maps with higher resolution. By contrast, according to your suggestion, we conduct additional experiments to examine the impact of lower-resolution inputs by further downsampling the TIL map to 100×. The results are shown below:

As can be observed from the table above, training on the lower resolution TIL map will lead to the decrease for survival prediction. Such experimental results suggest that excessive downsampling reduces the amount of useful information available for survival prediction.

[1] Saltz, Joel, et al. "Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images." Cell Reports 23.1 (2018): 181-193.e7.

Once again, we sincerely thank you for your thoughtful review and constructive feedback. If you have any further questions, we would be glad to respond. If you find our responses satisfactory, we would greatly appreciate your reconsideration of the score.

We sincerely appreciate the time and effort you have devoted to reviewing our submission. We have carefully prepared our rebuttal, aiming to address the valuable points you raised. Should you have an opportunity to review our response, we would be grateful for any additional thoughts or clarifications you may wish to share. Thank you once again for your thoughtful and constructive feedback.

Best regards,

The Authors

Dear Reviewer UaL5,

This is a friendly reminder that the reviewer–author discussion phase is approaching its end, and we notice you have not yet responded in the discussion thread. Please take a moment to engage with the authors' rebuttal by sharing your thoughts or follow-up questions. If your concerns have been addressed, kindly acknowledge that in the discussion thread. Before submitting the “Mandatory Acknowledgement,” please ensure that you have engaged in actual communication with the authors. Thank you again for your time and efforts.

Regards,

AC

Many thanks to the authors for their detailed and considerate rebuttal.

• W1: I thank the authors for showing that the model remains competitive even under embeddings different than PLIP.
• W2: In terms of technical contribution, I'd still like to maintain my original points. In my perspective, the three points that the authors have addressed still remain and the novelty is incremental compared to previous work.
• W3: The model being end-to-end and not working patchwise is consistently put forward as an advantage of the approach. While I agree that this is nice and practical, I find this coincidental with the fact that the TIL annotations are 50x downsampled, i.e. it is a limitation caused by the dataset. If full WSI-resolution TIL annotations were available, this direct end-to-end approach would not be possible, and I doubt that working at 50x downsampled resolution would lead to better results than using the full resolution and beat full-size foundation models. (I still appreciate that the model ends up having better survival prediction)

With these in mind, I find it difficult to push my score up from 4 to 5, and keep it as 4. I greatly appreciate the authors' rebuttal and offer my sincere thanks.