[ Response to Reviewer VjbS ]

We thank the reviewer for the valuable suggestions on our work. We have addressed the reviewer’s comments in our response below.

(Details of Table 1)

• Mean and Standard Deviation: The values in Table 1 represent the mean performance metrics, with standard deviations provided in Appendix C.1, Table 5. These standard deviations show the variation across 10 random data splits for each model, ensuring robustness and reliability. We have updated the manuscript to clearly refer to the appendix for details on standard deviation.
• Meaning of Bold and Underlined values: Bold values indicate the models with the best performance for each dataset. Underlined values indicate where the D-CAL measure is above 0.05, meaning that calibration is guaranteed.
• Statistical Significance of ConSurv’s Performance: To assess the statistical significance of the performance comparison, we conducted Welch’s t-test. In Table E, we present the findings where the improvement was statistically significant (i.e., p-value < 0.05). We have provided a detailed explanation and statistical analysis to substantiate our claim in the updated manuscript.

Table E. p-value for Performance Metrics.

(Details of Generating Positive/Negative Samples)

We construct the positive and negative samples based on the contrastive learning framework for tabular data [R8]. Specifically, we generate augmented versions $\tilde{x}^{(i)}$ of each data point $x^{(i)}$ by randomly selecting a subset of features (up to a pre-specified corruption rate) and replacing the value by sampling the corresponding feature's marginal distribution. The augmented version of the reference sample is used as the positive pair, and the augmented versions of the remaining samples in the mini-batch are used as the negative pairs. This corruption process is applied regardless of the feature types. We have included the details of generating the positive and negative samples in the updated manuscript.

(Additional Experiments: New Baselines)

In Table F, we have included a new baseline SupWcon which is a variant of our method where we replace $\mathcal{L}_{SNCE}$ with a supervised contrastive loss that directly weights negative samples based on the actual time differences of the comparable pairs, without considering their distributional properties of these weights [R11]. This can make weighting of negative samples sensitive to the scale of the time differences, potentially reducing the robustness of the contrastive learning loss. Notably, while weighted contrastive learning based on the difference in time-to-events show slight performance gains, the improvement is not significant when compared to ours. This suggests the importance of applying good inductive bias when contrasting samples, potentially because it encourages the model to focus on discriminating the survival prediction based on a proper distribution defined based on the differences in time-to-events.

Table F. Discrimination and calibration of survival models for ConSurv w/ $\mathcal{L}_{SupWcon}$.

(Temperature coefficient scale)

We thank the reviewer for insightful feedback regarding the use of $\sigma$ in our time-to-event contrastive learning framework. Here, we explain how $\sigma$ values affect model performance. When $\sigma$ is very small, the weight for time differences becomes excessively large, causing even minor time differences to be overstated and all negative samples to be treated equally. This leads to the model ignoring time-to-event information and behaving like standard contrastive learning. Conversely, with a very large $\sigma$, the weight for time differences becomes negligible, making the model insensitive to these differences. As a result, all samples receive similar weights, which also reduces the framework to standard contrastive learning. Our experiments ensure these findings, demonstrating that $\sigma$ must be chosen carefully. We observed that for contrastive learning to function effectively, $\sigma$ needs to be in a range where the scale of the weight applied to time differences does not excessively diverge from the model output. If the weight values become too extreme, either too large or too small, the model's performance is adversely affected. Thus, selecting an appropriate $\sigma$ that balances the impact of time differences is crucial for optimal contrastive learning performance.

(Implementation Details)

Min-max normalization was performed before the data was split into train, test, and validation sets.

(Clarification on Limitations)

We thank the reviewer for pointing out the need for clarification on limitations. We understand the importance of discussing the potential biases in detail and have incorporated your suggestions to make our proposals clearer in the updated manuscript.

---

Reference

[R8] D. Bahri et al., “Scarf: Self-supervised contrastive learning using random feature corruption,” ICLR, 2022.

[R11] Kerdabadi et al., “Contrastive learning of temporal distinctiveness for survival analysis in electronic health records,” CIKM, 2023.

Evaluating Calibration from Different Perspectives: We further provided the comparison in the model calibration by evaluating alternative calibration metrics -- Distributional Divergence for Calibration (DDC) [R6] and D-Calibration (D-CAL) [R7] -- that assess model calibration from a slightly different perspective. These metrics provide a more nuanced perspective on calibration quality, directly evaluating the alignment of predicted survival probabilities with uniform distributions. For the assessment of statistical significance of the performance gain in terms of model calibration, we have additionally included the p-values for the DDC in Table I. (Please note that the D-CAL test already incorporates p-values by its definition and thus is not considered for comparing the statistical significance of the performance gains.) Table I highlights that ConSurv achieves significant calibration gain over the benchmarks -- especially the ranking-based deep survival models -- across most of the datasets.

Table I. p-value for DDC

---

Reference

[R1] H. Uno et al., “On the C‐statistics for evaluating overall adequacy of risk prediction procedures with censored survival data,” Statistics in Medicine, 2011.

[R2] E. Graf et al., “Assessment and comparison of prognostic classification schemes for survival data,” Statistics in Medicine, 1999.

[R3] Z. Wang et al., "Survtrace: Transformers for survival analysis with competing events," ACM-BCM, 2022.

[R4] C. Nagpal et al., "Deep survival machines: Fully parametric survival regression and representation learning for censored data with competing risks," IEEE Journal of Biomedical and Health Informatics, 2021.

[R5] C. Lee et al., "Temporal quilting for survival analysis," AISTATS, 2019.

[R6] F. Kamran et al., "Estimating calibrated individualized survival curves with deep learning," AAAI, 2021.

[R7] H. Haider et al., “Effective ways to build and evaluate individual survival distributions,” JMLR, 2020.

[ Response to Reviewer VjbS ]

We appreciate the reviewer for the feedback and acknowledging our efforts in addressing the reviewer’s comments. Here, we would like to provide further clarification on the following points:

Normalization and Data Leakage: We apologize for any confusion caused by our initial explanation of the min-max normalization process. To clarify, we performed the min-max normalization after splitting the dataset, not before. This inadvertent miscommunication may have raised concerns about potential data leakage. We assure you that our method prevents any leakage, as normalization is conducted based solely on the training and validation sets. For verification, this procedure can be confirmed in the supplementary $`dataset.py`$ file that was submitted at the initial submission deadline.

Statistical Significance of the Performance Gain: We would like to start by emphasizing that our work introduces a deep survival model leveraging contrastive learning to enhance discriminative power without sacrificing calibration. Therefore, a fairer comparison showcasing our contribution would involve comparing our model with i) variations of our model trained solely with the NLL loss or with both the NLL and ranking losses, and ii) deep survival models utilizing traditional ranking loss. In this context, we have evaluated the statistical significance of our results based on the p-values calculated by the Welch’s t-test. As shown in Table G, ConSurv achieves superior discriminative power (while maintaining its calibration) over its variant trained solely with the NLL loss. The results are statistically significant except for the NWTCO dataset. On the other hand, our variant trained with both the NLL and ranking losses often loses its calibration power. Contrarily, when compared with the variant trained with the NLL and ranking losses, ConSurv significantly outperforms in terms of calibration power for all the evaluated datasets while providing gains in discriminative power (but not statistically significant for the NWTCO dataset) except for the SUPPORT dataset. These results are consistent with our motivation described in the Introduction. A similar trend can be observed when compared with traditional ranking-based deep survival models (i.e., DeepHit and DRSA). While the gain in discriminative power is sometimes not statistically significant, our method provides statistically significant improvements in terms of calibration power, as shown in Table G.

Table G. p-value for the variants of our method

Analysis of Time-Dependent Performance Metrics: While the CI and IBS provide valuable overall assessments of a given survival model (over the entire time horizon), these metrics cannot fully capture variations in model performance across different time points. To address this, we included time-dependent performance evaluations, namely the time-dependent C-index [R1] and the time-dependent Brier score [R2], in Appendix C.2, at three different time points (i.e., 25%, 50%, and 75% percentiles of time-to-events as in [R3, R4, R5]). It is worth highlighting that utilizing these time-dependent metrics may reveal subtle differences that might be obscured when using CI and IBS alone. Table H shows the number of statistically significant performance improvements over each survival model across different datasets. The improvement is considered significant if ConSurv provides statistically significant gains at at least one of the three time points. Again, we use the Welch's t-test to calculate the p-values for both time-dependent discriminative and calibration performance throughout the datasets. ConSurv offers superior calibration performance compared to ranking-based deep survival models (DeepHit and DRSA) and outperforms calibration-focused survival models (DCS and X-CAL) in discriminative power. Overall, compared to the deep learning baselines, ConSurv achieves statistically significant improvements in at least 4 out of 6 datasets, demonstrating its robust and consistent performance gains across various scenarios.

Table H. Number of Significant Performances Across Datasets

Hello reviewer VjbS,

Thanks for already engaging in discussion with the authors on this paper. The authors have responded in detail to your most recent comments. Please try to provide a response before the end of the author/reviewer discussion period (Aug 13 11:59pm AoE).

Thanks, Your AC

[ Response to Reviewer 1ZZS ]

We thank the reviewer for the positive feedback on our work. We have addressed the comments in the updated manuscript and provided a point-by-point response below.

(Technical Novelty about ConSurv)

ConSurv enhances the discriminative power of survival models by using a contrastive learning framework combined with a weighted sampling technique based on the similarity of event times. Instead of directly changing risk or survival predictions, this method focuses on distinguishing samples in a latent space according to their survival outcomes. It assigns weights to samples based on differences in their survival outcomes, focusing the learning process more on important samples. While Kerdabadi et al. [R9] uses a contrastive learning framework to time-to-event analysis, ConSurv is the first approach to use hard negative samples specifically to increase discriminative power for survival analysis. In addition, ConSurv achieves excellent calibration performance without using a separate loss function specifically for calibration [R6, R10]. This is achieved as the contrastive learning process inherently preserves good calibration while enhancing discriminative capabilities.

(Challenges of applying the proposed method to non-tabular data.)

Unfortunately, despite the potential of the contrastive learning framework utilized in our method, we were unable to find suitable non-tabular survival data for immediate application. To the best of our knowledge, the only available dataset is the METABRIC dataset, which contains whole slide images (WSIs) of breast tumors [R4, R5] with time-to-event information for each sample. However, these WSIs present a challenge: each sample contains a varying number of image segments (collected from different locations of a tumor). Addressing this would require either domain expertise to choose the correct tumor region of interest or a deep learning technique (such as multiple instance learning) to handle samples with varying numbers of image segments. As applying augmentation in this setting is not straightforward, it was beyond the scope of our current work and remains an interesting avenue for future research.

(Additional Experiment: t-SNE visualization)

We have included a t-SNE visualization for each of the real-world datasets we used in our experiments in the Global PDF (Figure 2). We show experimental results for FLCHAIN and SUPPORT, where performance improvements are significant. When using the NLL alone, the representations tend to cluster tightly in a narrow space, regardless of time. However, when applying SNCE (contrastive learning with a time weight) to NLL, representations of features with similar times are more likely to be positioned closer together. This shows that our model achieves superior performance.

(Clarification on Notations)

We thank the reviewer for pointing out typos and suggestions for better clarification. We have addressed those comments in the updated manuscript.

---

Reference

[R4] H. Xu et al., "A whole-slide foundation model for digital pathology from real-world data," Nature, 2024.

[R5] P. Mobadersany et al., "Predicting cancer outcomes from histology and genomics using convolutional networks," National Academy of Sciences, 2018.

[R6] F. Kamran et al., "Estimating calibrated individualized survival curves with deep learning," AAAI, 2021.

[R9] Kerdabadi et al., “Contrastive learning of temporal distinctiveness for survival analysis in electronic health records,” CIKM, 2023.

[R10] M. Goldstein et al., “X-CAL: explicit calibration for survival analysis,” NeurIPS, 2020.

[ Response to Reviewer Lfcy ]

We thank the reviewer for the valuable suggestions on our work. We have addressed the reviewer’s comments in our response below.

(Sensitive Analysis of the $\alpha$)

Introducing a margin prevents a comparable pair of samples with distant (unobserved) event times from being mistakenly treated as similar samples due to small differences between the observed event time of one sample and the censored time of the other sample within that pair. To show this effect, we have conducted an experiment using additional synthetic data by adapting the time-to-event generation process in DeepHit [R2] for both event times $T_i$ and censoring times $C_i$:

$T_i \sim \exp\left((10 x_{i1})^2 + 5 x_{i3}\right)$

$C_i \sim \exp\left((10 x_{i2})^2 + 5 x_{i4}\right)$

Here, we randomly generated 1,000 random samples, where a sample is censored when $T_i > C_i$.

In Figure 1 of the Global PDF, we compare time differences based on censoring times and the (unobserved) ground-truth event times. For each event sample, we compute the time differences between its event time and: 1) the (observed) censoring time of its corresponding comparable censored sample, and 2) the (unobserved) ground-truth event time of that same censored sample. Without a margin, censored samples with censoring times close to event times may be incorrectly treated as having similar event times, leading to very low weight assignments. To avoid this, we introduce a margin to exclude such samples when computing our contrastive loss. Overall, the margin in $|\tau_i-\tau_j| \geq \alpha$ helps prevent failures in accurately distinguishing risk between comparable censored samples.

(Sensitive Analysis of the $\beta$)

Table D. Impact of balancing coefficient $\beta$ on performance metrics.

F. Kamran et al. [R6] demonstrates that while ranking loss functions used in survival analysis focus on discriminative power, they often perform poorly on calibration metrics. Similarly, Qi et al. [R7] points out the independence of discrimination and calibration. They show that models with perfect discrimination can still have significant calibration problems. This suggests that finding a good balance is crucial for improving the model performance.

We investigated the relationship between discrimination and calibration, focusing on the impact of the balancing coefficient $\beta$. In Table D, additional experiments with $\beta=0.1$ and $\beta=10.0$ show that a very small $\beta$ (0.01) primarily focuses on NLL, resulting in good calibration. As $\beta$ increases, discriminative power improves while maintaining calibration performance. However, if $\beta$ is too large (beyond 1.0), it results in deviating from the concept of employing auxiliary loss, thereby leading to suboptimal discriminative and calibration performance. An optimal balance is achieved at $\beta=1.0$, performing well on both metrics. This highlights the need for hyperparameter searches to achieve optimal performance, as detailed in Appendix C.3.

(Details of Generating Positive/Negative Samples)

We construct positive and negative samples using the contrastive learning framework for survival data [R8]. SCARF addresses the challenge of tabular data augmentation while maintaining semantic integrity. Augmented versions $\tilde{x}^{(i)}$ of data points $x^{(i)}$ are created by selectively corrupting feature subsets using their marginal distributions. The augmented reference sample serves as the positive pair, while augmented versions from other mini-batch samples are the negative pairs. This permutation method avoids out-of-distribution issues common with noise injection described in Appendix B and is applicable to various data types. Further details have been provided in the updated manuscript.

(Laplacian Kernel)

We compared several kernel functions, especially the exponential and standard kernels. These kernels assign lower values to closer samples and higher values to farther samples. Our main goal, however, was to identify a kernel that not only maintains this distance-weighting property but also stabilizes at a finite value instead of increasing infinitely. Ultimately, we selected a modified version of the Laplacian Kernel, which has the property of having weights converge rather than diverge as the distance between samples increases.

(Clarification on Notations)

Each model function—$f$ (encoder $f_\theta$, hazard $f_\phi$) and $g$ (projection head $g_\psi$)—uses an MLP. Parameters were optimized using random search as detailed in Appendix E.2. We have maintained the representation as $z$ and updated the normalizing constant to $C$.

---

Reference

[R2] C. Lee et al., "DeepHit: A Deep Learning Approach to Survival Analysis with Competing Risks," AAAI, 2018.

[R6] F. Kamran et al., "Estimating calibrated individualized survival curves with deep learning," AAAI, 2021.

[R7] Qi et al., "Conformalized Survival Distributions: A Generic Post-Process to Increase Calibration", ICML 2024.

[R8] D. Bahri et al., “Scarf: Self-supervised contrastive learning using random feature corruption,” ICLR, 2022.

[ Response to Reviewer Lfcy ]

Once again, we would like to thank you for your invaluable feedback! We were wondering whether our response from Aug 7 has sufficiently addressed your comment. If you have any remaining comments, please let us know. We would be happy to do our utmost to address them!

[ Response to Reviewer Lfcy ]

Regarding the Comparisons with DRAFT and DATE:

We appreciate the reviewer's insightful comments regarding the comparison of ConSurv with DRAFT and DATE.

We opted to include DRAFT, not DATE, in our comparative analysis due to the following considerations: DRAFT posits a log-normal distribution for the underlying time-to-event process, $p(t∣x)$. This allows for a direct calculation of the survival function, $S(t∣x)$, facilitating a clear and equitable comparison using the performance metrics presented in our study.

Conversely, while DATE represents a significant advancement in deep survival models by directly generating time-to-event outcomes, it employs a generator that learns the underlying distribution, $p(t∣x)$, implicitly. Consequently, computing $S(t∣x)$ necessitates sampling a substantial number of samples, complicating the comparison against the reported performance metrics.

Therefore, we restricted our comparison to DRAFT as a deep learning counterpart that explicitly focuses on AFT models.

Regarding the comparison with CSD (Qi et al.’s Method):

We appreciate the suggestion to compare ConSurv with CSD (Qi et al.'s post-hoc calibration method), but we believe a direct comparison is not within the scope of our work. Our focus lies in improving discriminative power while preserving calibration, a distinct advancement from traditional ranking losses. CSD addresses a post-hoc approach to improve calibration of survival models, making a direct comparison less relevant.

Furthermore, Qi et al.'s work was published after our submission deadline and thus couldn't be included in our original analysis. However, we acknowledge its value and will incorporate a comparison in the appendix of our final version by applying their post-hoc calibration to existing ranking-based models.

The Connection Between Contrastive Learning and Model Calibration:

As the reviewer pointed out, Kamran et al. and Qi et al. have discussed the trade-off between discriminative power and calibration, where the trade-off arises because discrimination is typically assessed at an individual level, whereas calibration is evaluated at the population level. However, the parameter, $/lambda$, in the loss term proposed by Kamran et al. does not directly explain this trade-off. Instead, it controls the balance between two losses in the composite loss function, where each loss influences both the discrimination and calibration of the survival model.

To the best of our knowledge, there is no theoretical proof on why utilizing the ranking loss may lead to poor calibration. We hypothesize that directly modifying the output of survival models (whether the hazard function in DRSA or the PMF of event times in DeepHit) can bias the network towards optimizing the approximated C-index, potentially sacrificing accurate modeling of event time distributions, as conjectured by Kamran et al.

In contrast, our approach employs a contrastive learning framework that adjusts latent representations based on the similarity in the event times, implicitly promoting discriminative power without directly impacting the distribution of event times. Consequently, we were not able to observe a clear trade-off between the two loss functions, a phenomenon more commonly seen with traditional ranking losses.

Clarification on Notations:

We've taken steps to improve clarity by clarifying notations and providing comprehensive implementation details in both the appendix and this rebuttal. If you have specific areas that you feel still require attention, please point them out and we'll be happy to address them further.

Hello reviewer Lfcy,

The author/reviewer discussion period ends very soon (Aug 13 11:59pm AoE). It would be great if you could respond to the authors as they've taken the time to respond to your review.

Thanks, Your AC

[ Response to Reviewer RNpF ]

We thank the reviewer for the positive feedback on our work. We have included the details of censoring in the General Response Section.

(Missing Experiments on Unstructured Data)

Unfortunately, despite the potential of the contrastive learning framework utilized in our method, we were unable to find suitable unstructured survival data for immediate application. To the best of our knowledge, the only available dataset is the METABRIC dataset, which contains whole slide images (WSIs) of breast tumors [R4, R5] with time-to-event information for each sample. However, these WSIs present a challenge: each sample contains a varying number of image segments (collected from different locations of a tumor). Addressing this would require either domain expertise to choose the correct tumor region of interest or a deep learning technique (such as multiple instance learning) to handle samples with varying numbers of image segments. As applying augmentation in this setting is not straightforward, it was beyond the scope of our current work and remains an interesting avenue for future research.

---

Reference

[R4] H. Xu et al., "A whole-slide foundation model for digital pathology from real-world data," Nature, 2024.

[R5] P. Mobadersany et al., "Predicting cancer outcomes from histology and genomics using convolutional networks," National Academy of Sciences, 2018.