We thank the reviewer for providing interesting comments but are sorry that you did not agree with the other reviewers about the merit of our paper. To deal with your concern:

W1: … Lack of extensive discussion on the hardness of conditional calibration is a serious problem of this paper.

Thank you for stating your concern. However, we believe that this concern is mostly because this reviewer’s perspective is very different from ours and the other reviewers'. While our paper clearly relates to statistical issues, our goal was a method that survival analysis researchers can use to effectively achieve good calibration and maintain discrimination. That is why the main text de-emphasized theoretic proofs, and instead focused on intuitive explanations (using many real-world examples) of why our method should work. This stems from our concern that if our paper focused too much on conformal prediction and its history, the readers (mainly survival analysis researchers, because it is a tool for them) would lose interest.

Furthermore, Lei and Wasserman (2014) and Barber et al (2021) both note that conditional calibration is hard as they have shown that the finite sample guarantee for conditional calibration is impossible to achieve. This is why our paper does not try to provide any finite sample guarantee. Instead, we follow the route of many other researchers (Romano et al. (2019); Sesia and Candes (2019) [1]; Izbicki et al. (2020); Chernozhukov et al. (2021) [2]) by providing only an asymptotic guarantee for infinite samples. The main idea of this asymptotic conditional guarantee is that the construction of the post-conformal predictions relies on the original prediction, and we only hope for conditional calibration for the class of predictions that can be learned well (which correspond to the assumptions explicitly made in the paper, see also in W2).

Lastly, despite the hardness of conditional calibration and the limitation of our assumptions, our extensive experiments have shown the effectiveness and robustness of the method for 15 datasets and 7 baselines – to our best knowledge, no previous conformal prediction paper has provided such a broad empirical validation. While this conditional guarantee is hard in theory, our experiments certainly support it – which we view as a merit of our paper.

We again appreciate the suggestion. Towards reaching a broad audience, the revised manuscript will include a brief discussion about the hardness issue in the main text and also a new Appendix section to extensively discuss this.

[1] A comparison of some conformal quantile regression methods. Stat

[2] Distributional conformal prediction. PNAS

W2: … Theorem 3.2 actually shows nothing

We acknowledge that the consistent estimator assumption in Theorem 3.2 is weak. As mentioned earlier, our paper focuses on asymptotical guarantee – only providing conditional calibration in the class of predictions that can be learned well. This assumption strictly follows Assumption 3 in Sesia and Candes (2019); Assumption 2.3 in Izbicki et al. (2020); and Assumption 1 in Chernozhukov et al. (2021). Therefore, our Theorem 3.2 has the same practicality as those previous methods.

We also thank the reviewer for confirming that proving the conditional calibration is hard. Moreover, to our best knowledge, there are no results that can prove conditional calibration with more relaxed assumptions. Finally, note that our paper focuses on empirical performance more than theoretical results.

W3: … $\Gamma_M$ is a set of pairs of scalar values according to line 11 of Algorithm 1.

Thank you for noting this. Algorithm 1 uses this problematic expression (and also adds a direct comment behind this expression to explain) as we did not have a better notation to represent repeating a value by R times. The revised version will reduce this confusion by including an underbrace to express repeating a value.

W4: … If so, the equation between lines 537 and 538 is incorrect …

As shown under W7, here we prove that this theorem holds for $R=1$. Therefore, the equation between lines 537-538 is correct.

W5: … this paper assumes that the set $\Gamma_M$ does not have any tie in line 538 …

This “no tie” assumption is a technical assumption in conformal regression to make the proof. This assumption is easy to solve in practice because users can always add a vanishing amount of random noise to the scores to avoid ties.

W6: …the proof for censored data points does not argue $\rho_1$ and $\rho_2$

Thank you for pointing this out. The original proof only considers the upper bound to be adaptive and sets the lower bound to 0. We have included the revised derivation to incorporate an adaptive lower bound – see Figure 3 in the 1-page PDF. The revised derivation is almost the same as the original but has an extra term when breaking the joint probability into pieces. Fortunately, this modification does not compromise the validity of the conclusion.

W7: … Theorem C.1 completely ignores 𝑅, even though Theorem C.1 does not hold if 𝑅=1 …

In fact, the proof shows the theorem holds for $R=1$, as stated in lines 546-548: “if we do one sampling for each censored subject …, the above proof asymptotically converges…”. The repeating strategy is just to get a more accurate estimation for finite samples.

W8: …many implicit assumptions are used in lines 563 and 564.… during the proof of (i), a statement (i.e., an implicit assumption) similar to (i) is used in line 563.

Sorry about the confusion. The revised version will put the assumption (now in line 564) before the proof. Note however that the statement in line 563 is directly derived from line 562, so it is not an assumption.

W9: In the proof of Theorem C.2, the alleged proof of (i) on $𝑥_{𝑛+1}$ does not include any discussion on $𝑥_{𝑛+1}$.

Thanks for catching this typo! In lines 559-565, the conditions should be $x_{n+1}$, not $x_i$, which will be corrected in the revised manuscript.

We thank the reviewer for the insightful comments and suggestions. To address the concerns:

W1 & Q1: … would be helpful to have a similar visualization to illustrate the difference between CSD and CSD-iPOT.

Great suggestion! The revised version will include a side-by-side visual comparison of our method to CSD – see Figure 1 in the additional 1-page PDF. After the paper's acceptance, we will add two animated GIFs to the GitHub repo, to better visually illustrate the difference between our method and CSD.

W2: The technical novelty/innovation compared to CSD is perhaps somewhat limited, both methods employ the same general approach (post-processing methods which utilize conformal prediction).

It is true that both CSD and CSD-iPOT use the same general idea of conformal prediction. However, it is important to highlight that conformal prediction is an active area of research, with a variety of methods aimed at enhancing prediction coverage and calibration more effectively and efficiently.

Specifically, the novelty of this work stems from the unique design of the conformity score, and the downstream method for handling censored subjects (thanks to the conformity score design), which together provide a theoretical guarantee for the calibration.

W3: .. Overall, I find it quite difficult to judge how much added benefit CSD-iPOT actually would have compared to CSD in practice. Are there concrete examples where CSD leads to significant mis-calibration within a certain patient subgroup, which is addressed by CSD-iPOT? …

While our extensive experimental results did not find universal improvement, we did observe CSD-iPOT superior to CSD in 68/104 cases (significantly in 37 cases) for marginal calibration, and in 51/69 cases (significantly in 26 cases) in conditional calibration – see Table 2 in the paper. So in real applications, this method should be in your quiver of tricks to consider.

Furthermore, thanks to the reviewer’s great suggestion, we provide 4 case studies in Figure 2 of the 1-page PDF – concrete examples where CSD leads to significant miscalibration within certain subgroups (elderly patients, women, high-salary, and non-white-racial), but CSD-iPOT can effectively generate more conditional calibrated predictions. Furthermore, all 4 examples show that CSD’s miscalibration is always located at the low-probability regions, which corresponds to our statement (lines 208-212) that the conditional KM sampling method that CSD used is problematic for the tail of the distribution (low-probability regions).

Q2: … the top orange curve is always above the top blue curve in (e), whereas it is partially below the blue curve in (d)?

Thank you for your insightful comment. Yes, the blue curve is partially at the top in (d), intersecting the orange curve around 1.7 days, while the orange curve is consistently at the top in (e). This discrepancy arises from the discretization step used in our process, which did not capture the curve crossing at 1.7 days due to the limited number of percentile levels (2 levels at 1/3 and 2/3) used for simplicity in this visualization.

The post-discretization positioning of the orange curve above the blue curve in Figure 2(e) does not imply that the post-processing step alters the relative ordering of subjects. Instead, it reflects the limitations of using only fewer percentile levels. Note that other crossings, such as those at approximately 1.5 and 2.0 days, are captured. In practice, we typically employ more percentile levels (e.g., 9, 19, 39, or 49 as in Ablation #2), which allows for a more precise capture of all curve crossings, thereby preserving the relative ordering.

Q3: …both methods have a checkmark for "Discrimination guarantee Harrell’s" in Table 1?

We thank the reviewer for carefully examining this discrepancy between the statement and Table – yes, there is a typo in Table 1. For the proposed CSD-iPOT, the “Monotonic” should be a check, and “Discrimination guarantee Harrell’s” should be a cross. We are sorry for this mistake and will revise this and other typos in the revised version.

We thank the reviewer for the insightful comments and suggestions. To address the concerns:

W1: Missing related work in conformal prediction: the paper claims the method is based on conformal prediction but it is not mentioned and illustrated clearly in the main content …

The current method (detailed in Section 3) was intended to incorporate the detailed algorithm of conformal prediction so that readers with no conformal prediction background should be able to understand it, and the current Section 2.3 (lines 96-110) has summarized the related work on conformal prediction with censored data. However, thanks to the reviewer’s feedback, we will include a more detailed related work about standard conformal prediction.

W2 and Q1: How does the choice of percentiles affect the performance? Are there any empirical results that can illustrate the effect?

As noted in line 314, our Ablation #2 study explored the impact of predefined percentiles, with results presented in the Appendix. The short summary (lines 998-999) notes that the number of percentiles has no impact on the C-index and only a slight impact on marginal and conditional calibration. More details on the experimental settings and findings can be found in Appendix E.6.

As to the space complexity analysis, the number of percentiles indeed has an impact on that complexity. However, this impact is so subtle that in the big-O notation, it will be ignored. To expand more on this, Appendix E.5 establishes that the complexity of storing the conformity scores is $O(N \cdot R)$. After the method gets all the conformity scores, we will apply the percentile operation to all the conformity scores $\Gamma_{M}$ (as presented in Equation 4). The space complexity of this operation is just $O(|\mathcal{P}|)$ because there are only $|\mathcal{P}|$ scores to be saved (each for a unique percentile level) for later use. Therefore, the total space complexity is $O(N \cdot R + |P|) = O(N \cdot R)$, because $|\mathcal{P}|$ (a user-specific hyperparameter – here between 9-49) is significantly smaller than $N \cdot R$.

W3 and Q2: The discrimination properties of the method are unclear…

We thank the reviewer for carefully examining this discrepancy between the statement and Table! This is indeed a typo in Table 1. For the proposed CSD-iPOT, the “Monotonic” should be “check”, and “Discrimination guarantee Harrell’s” should be “cross”. The rest of the paper, including the statement in the main text and proofs in the Appendix, supports the above claim.

W4 and Limitation 1: The empirical exploration and validation of the group fairness is missing…

We thank the reviewer for bringing this up. Throughout the paper, we have considered the proposed conditional calibration metric also as a fairness metric. The resemblance to fairness metrics has been discussed multiple times in the paper, including lines 34-36 and 225-226.

To briefly illustrate their resemblances, the proposed conditional calibration score evaluates the calibration performance across all possible subgroups (and reports the worst score in all the subgroups). This aligns with some fairness definition that clinical decision systems should guarantee equalized performance (e.g., accuracy) across any protected groups.

However, we acknowledge that fairness has different definitions, and in this paper, we consider the proposed conditional calibration metric, $\text{Cal}_{\text{ws}}$, also as a fairness metric. We provide empirical results in Figures 4 and 11.

Q3: … How does the proposed method compare to the SOTA heterogeneity methods? Will it improve the performance?

The proposed method is a model-agnostic post-processing method, meaning it can be adapted to any heterogeneity survival methods, as long as they can generate individual survival distributions (ISDs). Our experiments have extensively demonstrated, using 7 SOTA heterogeneity models, that our approach can improve the marginal and conditional calibration without decreasing the discrimination performance.

As we noted, this approach can be very helpful in model development, as it allows researchers to simply seek models with superior discriminative abilities and subsequently apply the proposed method to improve calibration. This simplifies the model development process while ensuring robust performance across these key metrics.

Furthermore, the proposed method aims to maximize conditional calibration, a score that evaluates the calibration performance by considering the heterogeneity of all possible subgroups. Ideally, one would seek extreme heterogeneity – i.e., individual calibration – which means the prediction is calibrated conditioned on any possible combination of features. However, for each unique combination of features, there is only one realization and therefore is impossible to perform the evaluation. Instead, we reach a middle ground – conditional calibration. This idea is the motivation of our paper and is discussed in detail in Section 2.2.

We thank the reviewer for these wonderful comments and suggestions! Wrt your insightful concern:

The paper highlights that CSD-iPOT lacks theoretical guarantees for preserving Harrell's concordance …

Yes, we acknowledged this limitation in the paper. However, we argue this is not a big issue for two reasons:

1. While our method does not have a preservation guarantee for Harrell’s C-index, it does have this guarantee for two other discrimination metrics: AUROC and Antonili’s C-index, which are also (arguably more) commonly used in application studies. Due to their distinct nature, we do not know any method that is guaranteed to preserve all these three notions of discrimination (and suspect this might not be possible).
2. While our method is not guaranteed to preserve Harrell’s C-index, our extensive experimental results (on 104 comparisons) show that, in 84 cases, our method does not decrease the C-index. And for the 22 cases that it does decrease, none of them are significant. This really shows that this lack-of-formal-guarantee seems minor.

The description in lines 142-162 requires some improvements …

Thanks for the suggestion. This helped us to identify a possible misunderstanding. The proposed vertical transformation pertains solely to Eq 4. It does not apply to Eq 5, which transforms the inverse-ISD back to the ISD function (i.e., from Figure 2(e) to curves similar to Figure 2(a)). However, we realized that the position of lines 158-162 might cause this misunderstanding (now it is right after Eq 5); we will move this description ahead, to lead to a better presentation of the method.

I encourage the author(s) to also benchmark competitive baseline models that directly model event times …

We appreciate this suggestion. However, the proposed CSD-iPOT method requires the original survival models to be able to generate survival distributions (Section 3.1). If a baseline model can only generate a scalar value, it is not clear how to convert the scalar value into a distribution.

Also, because such a model only generates a time prediction (which is not a probability), it is also unclear what calibration means for time prediction.

However, if the reviewer has any insight on this, please let us know and we are happy to use your approach to benchmark these models.

Q1: … is there a hyperparameter that explicitly controls this trade-off?

No, there is no hyperparameter to control how much the CSD-iPOT process modifies the C-index. However, we do not consider this as an issue because (1) in 75/104 cases, the CSD-iPOT process did not affect the C-index, (2) for the remaining cases, the effect is not always negative: in 7/104 cases, CSD-iPOT improves the C-index.

Q2: It seems that CSD-iPOT achieves calibration lower than the empirical lower limit set by Kaplan-Meier. Are these the instances that result in an impact on the C-index?

Yes. The same columns in Figure 3 correspond to the same instances. The reviewer might think that there is an implicit trade-off – that increasing calibration is associated with decreasing C-index. However, this is not the case for our method: the level of calibration improvement is not associated with the level of C-index decreasing. For example, in the HFCR datasets with the GB baseline, our CSD-iPOT improves both C-index and calibration (Figure 3, left).

Q3: Why does CSD-iPOT struggle with both calibration and the C-index for models such as DeepHit?

Appendix C.4 discusses why our method is sub-optimal for these models (DeepHit and CQRNN), as indicated by lines 300-301 in the main text. In summary, this is because such models are often significantly miscalibrated, by implicitly assuming that, by the end of the predefined $t_{\text{max}}$, every individual must have had the event, and therefore their predicted survival distributions must drop to 0% at that time. In such cases, because CSD-iPOT can not have any intervention for the ending probability (0%), therefore, the ending position cannot be moved. Figure 7 shows an example of this issue, showing that the post-processed distribution has a sharp-dropping tail, which leads to miscalibration in those regions.

Q4: Are the calibration metrics obtained at different percentiles than those used for the post-processed ISD? ...

The percentiles for calibration evaluation are always set to be 10%, 20%, … 90%, as recommended by Haider et al. However, our Ablation #2 explores the impact of different predefined percentiles for the proposed CSD-iPOT. The short summary (lines 998-999) is that the number of percentiles has no impact on the C-index and a slight impact on marginal and conditional calibration. More details on the experiment settings and results can be found in Appendix E.6.

Q5: Why does CSD-iPOT improve marginal and conditional calibration models better than CSD? Is the difference due to how CSD-iPOT handles censored events?

Yes, that is exactly right!

For marginal calibration, because the CSD-iPOT shifts the survival probability values vertically (while CSD shifts the time prediction horizontally), it avoids the challenges of interpolation and extrapolation of the distribution, which is particularly problematic when the censoring rate is high and when KM ends at a high probability level.

For conditional calibration, CSD-iPOT is better than CSD because CSD-iPOT considers the heterogeneity of the features by sampling the event times from the individual survival distributions, while CSD samples from the conditional KM distribution do not consider the features.

The current paper discusses these two aspects in detail, in lines 206-215 and 194-205, resp.

I encourage the authors to discuss the limitations of their work, including any violations of modeling assumptions.

Thanks for the excellent suggestion. As mentioned above, the current paper discusses two limitations of our methods. However, the revised version will also include a discussion about the violations of assumptions.