Enhancing Multimodal Survival Prediction with Pathology Reports in Hyperbolic Space

Although this paper makes new investigations in computational pathology, such as hyperbolic representation and report integration, there are still substantial improvements to be made to get it published in well-known conferences like ICLR. My critical concerns are as follows:

• Writing: Some motivations are not elaborated clearly enough in the Introduction. For example, why did the authors choose to adopt MoME? From my understanding, CGM seems to be a better backbone for gene and WSI data, as shown in Table 1. Moreover, the explanation of the motivation behind using hyperbolic representation seems inefficient to readers, especially those unfamiliar with MERU (Desai et al., 2023). The authors are encouraged to improve it to make the paper self-contained and friendly to general readers.
• Experiments:
  • On page 2, line 99: The authors claim "our approach significantly enhances the ability of the model to capture the visual-semantic hierarchy". However, the paper fails to show the visual-semantic hierarchy the proposed model has captured. This is very important for readers to understand the mechanism and gain new insights, as the visual-semantic hierarchy is one main thing that this paper wants to demonstrate.
  • To ensure the general use of the proposed Hyperbolic loss, the authors are encouraged to validate it using a different backbone beyond the MoME. By doing so, the readers would know the proposed Hyperbolic loss is a generally-better tool, rather than being only effective in MoME.
  • On page 2, line 72: The authors claim "xxx a single pass, enhancing efficiency and xxx". I am curious about the efficiency of the proposed Tri-MoMe, e.g., how are the model efficiency and the computational efficiency? is the framework more efficient than existing ones?

The authors are encouraged to resolve the critical concerns above to make this work sounder to the computational pathology community.

Concerns with evaluation frameworks

• The authors use 5 fold-cross validation startegy without knowledge of tissue source sites to evaluate their model. This method has been shown to artificially inflate the performance of survival models (Kather) due to various demographic confounding factors ([1], [2]). Without the use of site-stratified splits, it is unclear how much performance can be attributed to confounding factors and actual signal. I request the authors to rerun their analysis with using site-stratified splits.
• The authors use CtransPath patch encoder. However, this model has been trained on TCGA and hence the same slides are used for pre-training and testing of the proposed method. 10+ foundation models for pathology have been proposed in the last year (Virchow 1/2 [3], GigaPath [4], UNI [5], CONCH [6], etc.), which are (1) significnatly better than CtransPath and are not trained on TCGA. I request the authors to (1) explain their rationale behind using CtransPath (2) try out one of the newer foundation models, which I believe should give a healthy improvement to the WSI only baselines.
• The authors optimize the hyperparameters of their model based on the validation set of their study (Table 3 for example shows that the best hyperparameters are chosen using the validation set). To avoid leaning towards test-set optimization of the model, I urge the authors to try out the WSI encoding part of their method on an independent test cohort, such as CPTAC-LUAD or CPTAC-UCEC.

Marginal gains over baselines

• The gain in performance with the proposed method seems small with respect to the standard deviations. I urge the authors to use statistical tests to support whether their method is significantly better than the baselines.

Missing baselines

• SurvPath (CVPR'24) [8] is an important baseline that is missing. Just like the authors' method, SurvPath uses gene-groupings. The authors know of this baselines as they cite them in lines 140-141.
• While it is commendable that the authors present KM curves to show patient stratification with their model, I would like to see KM curve comparisons with the next best baselines (for example CGM).

References

• [1]: Howard, Frederick M., et al. "The impact of site-specific digital histology signatures on deep learning model accuracy and bias." Nature communications 12.1 (2021): 4423. APA
• [2]: Vaidya, Anurag, et al. "Demographic bias in misdiagnosis by computational pathology models." Nature Medicine 30.4 (2024): 1174-1190.
• [3]: Vorontsov, Eugene, et al. "A foundation model for clinical-grade computational pathology and rare cancers detection." Nature medicine (2024): 1-12.
• [4]: Xu, Hanwen, et al. "A whole-slide foundation model for digital pathology from real-world data." Nature (2024): 1-8.
• [5]: Chen, Richard J., et al. "Towards a general-purpose foundation model for computational pathology." Nature Medicine 30.3 (2024): 850-862.
• [6]: Lu, Ming Y., et al. "A visual-language foundation model for computational pathology." Nature Medicine 30.3 (2024): 863-874.
• [7]: Xu, Yingxue, and Hao Chen. "Multimodal optimal transport-based co-attention transformer with global structure consistency for survival prediction." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023. APA
• [8]: Jaume, Guillaume, et al. "Modeling dense multimodal interactions between biological pathways and histology for survival prediction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. APA

• Limited Empirical Support for Pathology Report-Image Relationship Claims: Although the paper claims to effectively model the relationship between pathology report terms and corresponding images—specifically addressing that a single pathology term may map to multiple images and that pathology terms convey more generic concepts—this contribution is only discussed in the methods section. There are no experimental or visual results to substantiate this claim, leaving it unvalidated in the results section. Providing empirical or visualization evidence would strengthen this aspect of the paper.

• Incomplete Multimodal Relationship Modeling: While the paper addresses the relationship between text and pathology images, it does not appear to effectively capture the relationships between text and gene data or between images and gene data. Incorporating these additional interactions could enhance the model's ability to represent the complex dependencies across all three modalities and improve its overall performance in clinical settings.

• Limited Implementation Details: The paper lacks sufficient implementation information. It does not provide the experimental code, nor does it include details or examples on the preprocessing steps for pathology reports. This lack of transparency makes it challenging for others to replicate the study and validate the results, limiting the paper's practical impact and reproducibility.

In summary, although the problem and approach described by the authors are meaningful, additional technical details and experimental results are needed to fully demonstrate the effectiveness of their method. Providing more comprehensive implementation information and empirical evidence would strengthen the paper’s contributions and validate its claims.

Weaknesses

• This work misses comparison with current early fusion methods published in the last year: SurvPath in CVPR'24, PIBD in ICLR'24, and MMP in ICML'24. In addition, SurvPath, PIBD, and MMP propose new techniques for reducing complexity of cross-attention modeling in multimodal pathology-genomics data, which can be easily extended to text.
• How well do other methods perform with also text integration? There is missing investigation if method is performing better because it is including three modalities. All other comparisons have only one or two.
• How well does unimodal text and bimodal fusion (pathology+genomics, genomics+text, pathology+text) perform? There is missing investigation on the need to train with all three modalities within MoME. The bimodal fusion baselines should also be based on the MoME architecture with hyperbolic constraints. The pathology+genomics baseline is especially needed for apples-to-apples comparison with other comparisons that only use two modalities.
• How much does having a MoE-based architecture matter? There is missing investigation on comparing MoE design of Tri-MoMe with conventional Transformer architecture that can integrate tokens across long contexts in a straightforward manner. The problem in modeling long context may not matter as much with recent works like PIBD and MMP reducing WSIs into a bag of prototypes, which Transformers can easily perform cross-attention on.
• Not clear what prognostic information is captured from TCGA pathology reports using Tri-MoME. TCGA pathology reports contain very little information on observed morphological features. Features in TCGA pathology reports that would typically be prognostic would be patient demographic (age, sex), tumor size, tumor grade, and tumor stage - these features are more easily modeled as tabular attribute data that can be introduced into modeling via late fusion. A fair comparison would assess Tri-MoME and other models with pathology+genomics+clinical metadata (age, sex, grade, stage) fusion. The method feels too complex and there is missing investigation and analysis on what novel multimodal interactions are being learned by introducing a complex architecture.
• We should stop using CTransPath anymore in TCGA survival analysis due to data contamination. CTransPath is trained on TCGA. Using PLIP or CONCH as encoders make much more sense, especially with text integration. It is becoming increasingly more unethical to use pretrained models in such a manner with our growing awareness of data contamination.

Suggestions

I think the rating of this paper can be greatly improved if additional experimentation was performed in Tri-MoME for apples-to-apples comparison with other baselines (current SOTA with also text, Tri-MoME with pathology+genomics, Tri-MoME with clinical attributes, unimodal text survival analysis). Using CTransPath is problematic, but evaluation with a different encoder (preferably a vision-language encoder like PLIP/CONCH based on the theme of this work) would also greatly improve the soundness of this paper. Text integration is an interesting idea for improving multimodal cancer prognosis, but I don't think using TCGA pathology reports introduce any unique information that would also not be captured in clinical metadata. I think this aspect can be improved if the authors can show meaningful insights into pathology-genomics-text interactions being learned for survival analysis.