Leveraging Tumor Heterogeneity: Heterogeneous Graph Representation Learning for Cancer Survival Prediction in Whole Slide Images

Thanks for your constructive comment. Here are the responses to Weaknesses (W), Questions (Q) and Limitations(L).

W. The prototyes that are used in histology view are not learnt, potentially hindering the adaptability of the model since preselected prototypes may not capture the full variability or complexity of the data.

We fully agree with your comment. We also consider that prototypes selected before training based on prior knowledge indeed may struggle to capture the full variability or complexity of the data. Therefore, when designing the prototype extraction module (the Histology View (HV) module), we rely solely on the node categories provided by prior knowledge to give each prototype an initial preference. Then we use learnable shifts to obtain diverse prototypes for each category, enabling them to prefer on different factors within the category, including both previously known and unknown factors / phenotypes. Finally, under the guidance of these preferences, the prototypes to aggregate relevant information from the global features by learnable cross-attention. In summary, our prototype extraction process is learnable and highly flexible, and has the ability to capture various previously unknown factors / phenotypes that may contribute to patient outcomes.

We apologize for any misunderstanding caused by the description in our paper. We will revise this section in the camera-ready version to more clearly explain our prototype extraction process. Additionally, we have uploaded a detailed interpretability figure in the PDF attachment in "Author Rebuttal", which visualizes the attention preferences of multiple prototypes across different categories to further support our claims. If you are interested about it, please refer to "Author Rebuttal" for more details.

Q. Compare with PANTHER.

Thanks for the advice. We compared PANTHER with our model. Additionally, we compared our model's Histology View (HV) module (prototype extraction module) to ensure the fairness of the prototype extraction module comparison. We tested two pooling methods for the HV module, including the mean pooling method used in the ablation experiments in the main paper (corresponding to the "without (w/o) HV" row in Table 2 in the main paper), as well as the concat pooling method used by PANTHER (PANTHER performs pooling by concatenating all prototypes along the channel dimension). The experimental results are as follows:

The learnable prototype extraction method (HV module) outperforms PANTHER on most datasets. In addition, we observed that PANTHER's performance surpassed all comparison models on the PAAD dataset. We attribute this to the relatively small sample size of the PAAD dataset (208 cases), which may lead to overfitting. Therefore, PANTHER, which extracts prototypes in an unsupervised way based on priors and has a minimal number of learnable parameters, demonstrated superior results on the PAAD dataset.

L. Apart from the comparisons with SOTA methods, it would be interesting to shift the focus also to the interpretability of the model.

Thank you for your interest in the interpretability of our model. We have expanded the figure to visualize the preferences of each prototype and the tissues they focus on, to better demonstrate the interpretability of our model. if you are interested about it, please refer to "Author Rebuttal" for more details.

Once again, thank you for your careful review and valuable comments on our paper. If you are satisfied with our response, would you kindly bump up your score?

I'm sorry for the reviews submitted for another paper. I update the reviews here.

This paper proposes ProtoSurv, a heterogeneous graph model for WSI survival prediction. ProtoSurv is driven by data and incorporates pathological domain knowledge. Specifically, ProtoSurv consists of three modules 1) a multi-layer GCN to learn the structure representation of WSIs, 2) a prototype representation method to learn pathological priors, 3) a prior guided fusion method to aggregate structure view features and pathological multi-prototypes. The experiments on TCGA verify the effectiveness of ProtoSurv. The motivation of this paper sounds good. However, the model lacks novelty and in-depth exploration. In addition, ProtoSurv requires additional labels to train a classifier, leading to unfair comparison.

Soundness: 3: good Presentation: 3: good Contribution: 2: fair Strengths: This paper is easy to follow. The motivation sounds good. This paper proposes ProtoSurv, which deciphers intratumoral tissue heterogeneity using a heterogeneous graph and incorporates prior knowledge of prognostic tissue types into the prediction process. The experiments on TCGA verify the effectiveness of ProtoSurv. Weaknesses: The design of the main modules lacks novelty and in-depth exploration. The Structure View (SV) is a type of multi-layer feature aggregation that has been widely studied and may be highly influenced by the selection of layers to extract features. The Histology View (HV) is a type of feature shifting which has also been widely studied. ProtoSurv requires additional labels to train a classifier, leading to unfair comparison. Questions: More fine-grained ablation experiments and comparisons are helpful: 1. How about only using the last or last two layers to extract features in SV? 2. How about sharing the learnable parameter for each category in HV? 3. How about the comparisons of FLOPs and the number of parameters?

Thanks for your constructive comment. Here are the responses to Weaknesses (W), Questions (Q) and Limitations(L).

W1. The design of the main modules lacks novelty and in-depth exploration

Novelty. We fully agree with your summary of the modules within our model. However, the crucial aspect is that we construct a framework capable of leveraging intratumoral tissue heterogeneity and utilizing node types to introduce pathology priors into the model. To our knowledge, this is novel in the computational pathology literature.

We also introduced a more suitable prototype extraction method tailored for computational pathology tasks. In existing prototype-based computational pathology networks, prototypes are often extracted simply based on fixed cluster centers, which may struggle to capture the full variability or complexity of the features. By contrast, in our approach, We design learnable shifts to extend each prior-based prototype into multi-prototypes with different preferences. These prototypes focus on interactions with different tissues and learn from previously unknown factors or phenotypes that may contribute to patient outcomes.

Additionally, we provide an interpretability figure illustrating the interactions between prototypes from different categories and the global tissue, which allows for a better exploration of the global tissue interactions discovered by the model. We have expanded the interpretability figure in the PDF attachment in "Author Rebuttal", if you are interested about it, please refer to "Author Rebuttal" for more details.

In-depth exploration. For the finer-grained ablation experiments and comparisons you suggested, we add ablation studies in the following sections. If you believe there are other areas that require in-depth exploration, we are willing to include additional experiments.

W2. ProtoSurv requires additional labels to train a classifier, leading to unfair comparison

We used classifier-obtained node types to inject pathology priors into the model, aiming to explore the benefits of incorporating domain priors. The experimental results indicate that the model indeed improved with the assistance of these domain priors. In the ablation experiments shown in Table 4 of the main paper, we demonstrated that when using publicly available classifiers or simply using K-means to obtain node categories, our framework still achieved better results than the compared models. This indicates that in the fair comparison without the advantage of pretrained classifier node types, our framework's ability to handle tumor heterogeneity still leads to superior prediction performance compared to baselines.

Q1. How about only using the last or last two layers to extract features in SV? Thanks for the advices. We tested using only the last and the last two layers to extract features in SV. The experimental results are as follows:

Although the optimal results varied across different datasets, overall, models that used more layers to extract features achieved better results.

Q2. How about sharing the learnable parameter for each category in HV?

We greatly appreciate your advice. Sharing the learnable parameters for each category can significantly reduce the model's parameter. This helps explore the scalability potential of the model. Here are the results of sharing the learnable parameters for each category.

We show the optimization of computational requirements brought by sharing the learnable parameters in Q3.

Q3. How about the comparisons of FLOPs and the number of parameters?

Follow your advice, we evaluate the model's inference time, floating point of operations(FLOPs), model parameters, and maximum GPU memory usage. We use a WSI which contains 32,625 patches as input. The computation time is measured using a Nvidia RTX 3090 GPU.

We included PatchGCN for comparison. We additionally test ProtoSurv-tiny under a reduced parameter configuration (prototype dim = 256, hidden dim of SV and HV = 64, prototypes per category = 4), to evaluate the performance degradation of our architecture with fewer parameters and its scalability for more limited hardware.

Here are the survival prediction results of ProtoSurv under the reduced parameter configuration.

Once again, thank you for your careful review and valuable comments on our paper. If you are satisfied with our response, would you kindly bump up your score?

Thanks for your constructive comment. Here are the responses to Weaknesses (W).

W1. Having a fixed set of hand crafted node types and a fixed number of multi-prototypes is limiting, and restricts the model’s ability to learn from previously unknown factors / phenotypes that may contribute to patient outcome.

We fully agree with your comment. We consider that simply extracting prototypes for each node category based on priors might indeed restrict the model's ability to learn from previously unknown factors / phenotypes that may contribute to patient outcome.

Therefore, when designing the prototype extraction module (the Histology View (HV) module), We made the following efforts to ensure the model is not constrained by node types, and to enhance the model's ability to learn from factors/phenotypes beyond the priors:

1. We rely solely on node types provided by prior knowledge to give each prototype an initial preference, rather than using them as constraints for the prototypes.

2. Then we use learnable shifts to obtain diverse prototypes for each category, enabling the model to prefer on different factors within the category, including both previously known and unknown factors / phenotypes.

3. Finally, under the guidance of these preferences, the prototypes to aggregate relevant information from the global features by learnable cross-attention.

In summary, our prototype extraction process is learnable and highly flexible. The node types only provide an initial preference for each prototype, rather than a constraint. The fixed number of multiple prototypes obtained through offsets increases the diversity of prototypes within each category, encouraging the model to have more varied preferences. Finally, through learnable cross-attention, each prototype extracts relevant features from the global context (without being limited by node types), further enhancing the model’s ability to learn from previously unknown factors/phenotypes that may contribute to patient outcomes.

We apologize for any misunderstanding caused by the description in our paper. We will revise this section in the camera-ready version to more clearly explain our prototype extraction process. Additionally, we have uploaded a detailed interpretability figure in the PDF attachment in "Author Rebuttal", which visualizes the attention preferences of multiple prototypes across different categories to further support our claims. If you are interested about it, please refer to "Author Rebuttal" for more details.

W2. Evaluation benchmarks should include popular aggregation baselines, including ABMIL (used by UNI authors for benchmarking) and TransMIL.

We include additional experiments with popular aggregation baselines, the results are as follows:

W3(1). Ablation study: Use UNI features directly for prototype learning, without considering node types.

Following your advice, we supplement the ablation experiments here. In this ablation experiment, we used 8 cluster centers obtained from the K-means algorithm as the initial values of each prototype (corresponding to the process of eq (4) in the main paper), and performed prototype learning directly without considering node types. Here are the experimental results:

W3(2). Ablation study: Removing graph network, or replacing it with a transformer.

We performed an ablation experiment on removing the graph network in the "without (w/o) SV" row of Table 2 in the main paper. However, we acknowledge that this experiment is still insufficient. Following your advice, we supplement the ablation experiments here. We remove the graph network (Structure View module (SV)) while retaining the prototype extraction module (Histology View module (HV)), and test the performance of the prototypes extracted by HV under different pooling methods. We test mean pooling by calculates the average value of all prototypes (HV(mean pooling) row), and concat pooling by concatenating all prototypes along the channel dimension(HV(concat pooling) row).

Here are the experimental results:

Once again, thank you for your careful review and valuable comments on our paper. If you are satisfied with our response, would you kindly bump up your score?

Thanks for your constructive comment. Here are the responses to Weaknesses (W), Questions (Q) and Limitations(L). Due to word limit, we omit many details. If you have any further questions, we would be willing to response.

W1. This is not a heterogeneous graph neural network.

We describe our model as a heterogeneous graph neural network in the paper, because we were indeed inspired by the optimization of heterogeneous graphs of aggregating information from global nodes [1]. Similar to them, each node in our model has a category which participates in guiding the global feature extraction. Therefore, we cautiously feel that it is appropriate to define the model as a heterogeneous graph-based model.

However, we believe that the dual-stream architecture indeed better captures the model's characteristics. Each module has its own focus and does not need to be collectively described as a heterogeneous graph. We will revise the description to highlight the characteristics of its dual-stream architecture.

[1] Li et al. Finding global homophily in graph neural networks when meeting heterophily, ICML2022.

W1(1). Compare with Patch-GCN and prototype-based models.

Here, we provide a more detailed ablation experiments about prototypes. Additionally, as a complement comparison to the prototype-based model, we reference PANTHER [2], a SOTA prototype-based unsupervised model, for comparison. We test mean pooling and concat pooling of Histology View (HV). Same as PANTHER, concat pooling concatenate all prototypes along the feature dimension.

Here are the results:

[2] Song et al. Morphological prototyping for unsupervised slide representation learning in computational pathology, CVPR2024.

W2. Computational requirements and scalability

Computational requirements. We evaluate the model's inference time, floating point of operations(FLOPs), model parameters, and maximum GPU memory usage. We use a WSI which contains 32,625 patches as input. The computation time is measured using a Nvidia RTX 3090 GPU. We additionally test ProtoSurv-tiny under a reduced parameter configuration (prototype dim = 256, hidden dim of SV and HV = 64, prototypes per category = 4).

Here are the results of ProtoSurv-tiny.

Scalability. The HV and SV modules, as well as the process of extracting prototypes, are completely decoupled, and can be computed in parallel.

W3. The ablation on the PGF module.

Here, we include a supplement ablation study for the PGF module. In this ablation study, we remove the fusion between the two views and instead concatenated directly along the patch dimension.

W4. An interesting Figure 10 in the Appendix.

We greatly appreciate your interest of Figure 10. As per your suggestion, we will reorganize the camera-ready version to move it to "Experiments" section. In addition, we have expanded the figure, please refer to the PDF attachment in "Author Rebuttal" for more details.

W5. SV doesn't simulate viewing at multiple magnification.

GNNs extend the receptive field, and each GNN layer provides different neighborhood information. In SV, all output from each GNN layer are concatenated, which aggregate features with varying hops of neighborhood and different receptive fields. We full agree with you that it strictly provides multi-hop neighborhood information rather than multi-scale information. We will revise the description to make it more precise.

L1. Discussion on how errors in classification might impact the overall performance.

To minimize the impact of classification errors on the overall performance, in the HV module, we rely only on the node categories provided by the classifier to delineate an initial range. We calculate the average value of the patch features within the category range as the initial prototype, providing an initial preference for the aggregation of each prototype (pathology prior injection).

In the main paper's Table 4, we demonstrated the robustness of our model using ablation results with publicly available classifiers and clustering methods. To further illustrate this point, we randomly generated categories for 20% and 30% of the nodes. The experimental results are as follows:

Once again, thank you for your careful review and valuable comments. If you are satisfied with our response, would you kindly bump up your score?