Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Prediction

Thanks for your constructive comments and suggestions, they are exceeding helpful in improving our paper. We have carefully incorporated them in the revised paper. In the following, your comments are first stated and then followed by our point-by-point responses.

Q1) Clarity. The clarity of the PIB and PID explanations, particularly in Section 3, needs improvement for broader appreciation. The introduction of Markov chains confuses the paper more - Perhaps better to remove it.

In response to your valuable feedback, we have revised Section 3 of the paper. The modifications include removing the introduction to Markov chains, emphasizing the changes made by the new approach more straightforward for conciseness and clarity.

Q2) Discriminative assumption. 1. The suitability of a "discriminative" approach for survival prediction. 2. The proposed approach seems only applicable to NLL loss. What if more common loss functions, such as Cox PH or rank losses, were to be used that do not rely on such discriminative hypothesis?

1. We agree that survival prediction problems present greater complexity than diagnosis. In practical scenarios, there may exist deciding biomarkers within different risk bands, which leads to the “discriminative” method having demonstrated good performances in many cases [1,2]. Although some instances related to different risk groups may exhibit some overlap, these overlapping instances may not significantly contribute to distinguishing risk bands. Similarly, the same situation of overlapping can be observed in subtyping/grading as well. Therefore, our method, through the use of prototypes, excels in identifying discriminative features that are specifically associated with the positive risk level. Hence, discriminative assumption may not be a weakness but an advantage for biomarker discovery of different risk levels.

2. Indeed, the prototype learning in the current approach primarily relies on cross-entropy(CE)-like loss. This is because the prototypes are constructed based on information bottleneck theory, in which the first term of the derivation in Eq. (8) is formulated as a form of cross-entropy. Fortunately, NLL loss, a CE-like loss, makes it feasible to employ IB for survival prediction task.

   Although the prototype component is based on CE-like loss, the task loss in our framework is still flexible and can be easily switched to Cox or rank loss. Furthermore, even in rank-based losses, which focus on relative comparisons between samples, discriminative information is still needed. Although not directly involving classification, the model needs to learn the differences between samples to correctly rank them. Hence, the assumption of PIB may not conflict in such scenarios.

   Despite the discriminative hypothesis, this work first attempts to directly model the bag-level representations for a bag structure through the probabilistic model in survival prediction task. There is undoubtedly significant value in expanding this approach to various hypotheses like rank-based hypothesis. Hence, further investigation is still warranted in the future.

REFERENCES

[1] Echle A, Rindtorff N T, Brinker T J, et al. Deep learning in cancer pathology: a new generation of clinical biomarkers[J]. British journal of cancer, 2021, 124(4): 686-696.

[2] Wiegrebe, Simon, et al. "Deep Learning for Survival Analysis: A Review." arXiv preprint arXiv:2305.14961 (2023).

Thank you for your encouraging words and constructive comments. We sincerely appreciate your time in reading the paper. In the following, your comments are first stated and then followed by our point-by-point responses.

Q1) The Prototypical Information Disentanglement (PID) module could benefit from improved clarity in its description.

Regarding the clarity of the PID module, we have revised the corresponding section 3.3 on Page 6 in the manuscript to enhance the understanding for readers.

Q2) What is the significance of the red arrows in Figure 1, particularly within the context of the PID module?

The red arrows in Figure 1 signify the computation of the joint prototypical distribution, where each arrow connects to the positive prototype of a sample. The calculated joint distribution gives guidance for extracting modality-common and modality-specific information.

Q3) How does PIB save the computation? Please make quantitative analysis.

Thank you for your feedback. For the computation efficiency of PIB, it can be known that directly employing VIB would lead to intractable bag-level distribution $p(z|\bf{x})$ due to the large number of individual instance distributions within a bag $\bf{x}$, based on the analysis in the second paragraph on Page 5. The proposed PIB can simplify this issue by modeling bag-level distribution with prototypes.

We can coarsely quantify the reduction in computational complexity compared to the original VIB. Assuming the sampling frequency is $K$, the number of samples per iteration is $N$, and there are $M$ instances of $D$-dimension in a bag, the computational complexity is $O(K\cdot N\cdot M\cdot D)$. However, in PIB, we only need to sample from prototypes. Assume that the number of prototypes is $P$, the complexity becomes $O(K\cdot N\cdot P\cdot D)$, where $P << M$. For example, P is 8 in our work, while M is usually larger than 10,000.

Thank you for your encouraging words and constructive comments. We sincerely appreciate your time in reading the paper. In the following, your comments are first stated and then followed by our point-by-point responses.

Q1) A “naive” baseline. The evaluation could be improved by adding a “naive” combination of the risk scores from genomics and histology. i.e. Combining individual modality predictions using CoxPH.

Thank you very much for this insightful comment! We follow your suggestion and combine the predictions of SNNTrans and CLAM-MB as a “naive” baseline. The results are added in Table 1, Page 8, and are also shown in the table below.

From the results, it's clear that the "naive" combination approach at the prediction level yielded unsatisfactory results. The main reason behind this suboptimal performance is that this method focuses solely on single-modal learning, failing to extract relevant common information between modalities. This further underscores the superiority of our approach, which, through information disentanglement, eliminates redundancy between inter-models and effectively captures both specific and common information.

Q2) Kaplan-Meier analysis. Please state how cut-offs were selected in the manuscript.

Thank you for your suggestion. We have provided more details about the Kaplan-Meier analysis in our manuscript. In our work, we employed a five-fold cross-validation evaluation, getting one model for every data fold. For every validation set, we determined the cut-offs by taking the median of the model-predicted risk values. Patients with a risk higher than the median are assigned as high risk, and patients with a risk lower than the median as low risk. In addition, in response to the suggestion from reviewer G86X, we have also reported the median survival months for each risk group on Page 8.

Q3) The hyperparameter choices. How hyperparameters were selected with the dataset. As cross-validation is used to generate the evaluation results, to what extent were the hyperparameters selected using the evaluation datasets and is there risk of selection bias as a result?

Thank you for bringing up this important point. Due to the limited size of datasets, like many other methods, we chose cross-validation as the assessment approach, allowing us to effectively leverage the available data and enhance the reliability of our evaluation.

In our study, we adopted a careful approach to address the potential risk of selection bias in hyperparameter choices. To begin with, we defined the hyperparameter search space based on the commonly used ranges in information bottleneck methods like VIB[1] and MIB[2]. During the selection process, we implemented a grid search strategy. For each hyperparameter, while keeping others fixed, we conducted a five-fold cross-validation, selecting the hyperparameter that exhibited the best average performance across all validation sets. Furthermore, to further mitigate selection bias, these hyperparameters were selected on multiple datasets, ensuring the generalizability of the chosen configuration. By employing this approach, we aimed to minimize the risk of selection bias.

we have incorporated the introductions on this aspect into Appendix C.2 on Page 19.

REFERENCES

[1] Alemi, Alexander A., et al. "Deep Variational Information Bottleneck." International Conference on Learning Representations. 2016.

[2] Federici, Marco, et al. "Learning Robust Representations via Multi-View Information Bottleneck." 8th International Conference on Learning Representations. OpenReview. net, 2020.

Thanks for your constructive comments and suggestions, they are greatly beneficial in enhancing the quality of our paper. We hope our answers will address the concerns and clarify the contributions of the paper. In the following, your comments are first stated and then followed by our point-by-point responses.

Q1) Limited findings w.r.t. clinical application / multimodal interpretability

Thank you for your feedback and suggestions! In order to present the concerns in a more organized manner, we will state our response point-by-point.

Contribution. We'd like to emphasize the contributions of our method, as highlighted by Reviewers bKh7, HNZs, and G86X. Our approach achieves innovation by reducing redundant information within and between modalities for survival prediction through the information bottleneck in a quite novel prototypical method. Methods like MOTCat and SurvPath, which utilize co-attention mechanisms, focus specifically on gene-related instances of WSIs, emphasizing modality-common information. They failed to address the issues that we’re concerned about in the following aspects. First, these approaches ignored the distinct value of modality-specific information. Second, helpful instances in a bag only occupy an extremely small portion, which means massive intra-modal redundancy. Without explicit redundancy removal, such as MOTCat and Survpath, useful information will be flooded. Lastly, they cannot directly model at the bag-level, as noted by Reviewer G86X, "probabilistic modeling on bags opens up new avenues for probabilistic approaches for CPath, which have been under-explored thus far.”

Interpretability. For local interpretability of individual patients, we have visualized similarity scores of instances to each prototype of risk level and presented top-3 relevant patches and biological pathways, as well as top-3 irrelevant biological pathways. Through analyzing these instances, we may discover biomarkers of various risk levels. This is because prototypes are designed to be discriminative, which enforces selecting instances that are most distinctive for each risk level, potentially corresponding to biomarkers for different risk levels. In addition to what we have done, another potential way is to analyze the modality-specific features extracted from PID using CAM-based approaches, which will facilitate the discovery of unique biomarkers within each modality.

Since prototypes represent the distribution for each risk band, they inherently possess a global nature. As a result, the global interpretability can be demonstrated in the following aspects. First, by collecting and organizing instances with high similarity to prototypes across a large patient cohort, we can assess cohort-level disease survival. Second, in PID, the joint distribution of histology and genomic data containing modality-common information, contributes to identifying paired instances related to both modalities and further establishing their relationship between patches and pathways. This also helps to understand their collaborative mechanisms.

Clinical applications. We truly appreciate your suggestions and believe that the aforementioned way to interpret the model has the potential to provide some clinical value. However, the investigation of clinical applications requires collaborative efforts from experts across disciplines, such as pathologists, statisticians, AI experts, etc. For example, interpretability in clinical applications demands extensive review of pathology slides by pathologists and continuous enhancements in algorithms by AI engineers[1-3]. Therefore, it definitely takes a lot of time and effort in development and elaboration, which has gone beyond the scope of the current work. This work still primarily focuses on technical innovation in methodology. We acknowledge the importance of clinical applications, and we will continue to explore this area and hope to report the findings soon.

REFERENCES

[1] Hosseini, Mahdi S., et al. "Computational Pathology: A Survey Review and The Way Forward." arXiv preprint arXiv:2304.05482 (2023).

[2] Berbís, M. Alvaro, et al. "Computational pathology in 2030: a Delphi study forecasting the role of AI in pathology within the next decade." EBioMedicine 88 (2023).

[3] Lu, Ming Y., et al. "AI-based pathology predicts origins for cancers of unknown primary." Nature 594.7861 (2021): 106-110.

To the authors,

Thank you for your detailed comment in addressing the points raised. From carefully re-reading the updated manuscript, new experiments, and references, here are my comments:

Response to Q1) Limited findings w.r.t. clinical application / multimodal interpretability:

Contribution: I agree with the authors and with Reviewer G86X that probabilistic approaches for pathology is a relatively under-explored but promising direction. Regarding notions of "modality-specific/shared" and "intra-/inter-modal redundancy" and comparisons to previous works):

• "Modality-shared information": This claim is well-supported. In the reviewer's assessment, shared information is modeled via aggregating patch-level pathology feature embeddings based on their similarity with genomic pathway embeddings.
• "Modality-specific information": This claim is more nuanced, and slightly disagree that SurvPath and MOTcat "ignored the distinct value of modality-specific information". In MOTcat, following computing the OT plan from pathology to genomics (similar to co-attention), genomic pathway features are still explicitly introduced in the risk prediction layer. In SurvPath, genomic-to-genomic relationships are still modeled using Transformer attention and fed into the risk prediction layer. The authors should clarify that genomic features (though not explicitly modeled to have independence with pathology) are still used for survival prediction. Pathology features (without genomic guidance) are not used, supporting the authors' claim.
• "Intra-modal redundancy": I disagree more with this point. Redundancy in pathology-specific modality is performed via co-attention / OT, where pathology embeddings are pooled based on their similarity to genomic pathway embeddings. Using the processed genomics data made available by SurvPath, at most only 331 pathway tokens are used. As pathways are somewhat prototypical and semantic in describing a unique biological process (different from repeating histology patterns for pathology), there is less motivation to mitigate redundancy here as the tokens are relatively semantically-dense and much lower than bag sizes for WSIs.
• "Inter-modal redundancy": This claim is well-supported.

Interpretability and Clinical Application: I still stand by my original point that the attention heatmaps are difficult to interpret and draw conclusions from. To be more specific:

• "We have visualized similarity scores of instances to each prototype of risk level and presented top-3 relevant patches and biological pathways, as well as top-3 irrelevant biological pathways":
• • As the technical contributions of PIBD are in mitigating intra/inter-modal redundancy and emphasizing modality-specific/shared features, it would be useful to understand which specific features in pathology and genomics are captured, and which features are shared across pathology and genomics modality. At the moment, the interpretability in Figure 6 and 7 are similar to that of MCAT / SurvPath / MOTcat in visualizing pathology-genomic correspondences despite having a very different approach. What is modality-specific/shared?
• • What does "Similarity Scores of Pathways" show? Font sizes are too small. Common practices for reporting pathology images such as magnification per pixel are missing. Variables such as $t$ and $c$ are not defined within the figure.
• "Since prototypes represent the distribution for each risk band, they inherently possess a global nature...":
• • With number of prototypes $P=8$, what are the these prototypes learned for pathology and genomics? For pathology, do they relate to a distinct or important morphology? For genomics, do they reflect general "tumor-suppressing" or "immune-activating" pathways (which pathways get grouped together)? For multimodal interpretability, which prototypes are shared and which are conserved? In Figure 6 and 7, it is not clear what each prototype is. Figure 4 is also not very informative on which prototypes are learned. From existing prototype-based works in pathology like H2T [1], I believe the authors should be able to perform a more detailed interpretation of prototypes.

Empirically, this method still performs well. My concern here is that the inductive biases that PIBD solves (intra-/inter-modal redundancy) are not properly investigated and validated. As pathology has many domain-specific challenges for learning representaitons but also unique opportunities for proposing new methods that address domain-specific inductive biases, such inductive biases such as intra-/inter-modal redundancy should have more validation to strengthen its contributions in both ICLR and in pathology.