Conformalized Survival Analysis for General Right-Censored Data

We appreciate your engagement, helpful suggestions, and valuable feedback. In what follows, we respond to the reviewer’s concerns in detail.

Weaknesses

The importance of the right-censoring setting

We kindly refer the reviewer to GLOBAL RESPONSE: CONTRIBUTIONS (Importance of reliable survival analysis under general right-censored data).

The novelty of our work in view of Gui et al. (2024)

We kindly refer the reviewer to GLOBAL RESPONSE: CONTRIBUTIONS (Key novelty) for a discussion of the significant theoretical and technical advancements in our work.

Using only a subset of the calibration set

We understand that the idea of using a subset of the calibration points to enhance performance is not intuitive at first. However, this approach can greatly improve statistical efficiency compared to the naive method that uses all the calibration points. Our work focuses on carefully selecting a subset of calibration points to produce more informative LPBs while maintaining validity. In Proposition 3.1, we characterize the condition under which such a selection of a subset of points (those whose $e=1$) will provably improve performance. Loosely speaking, this is because only a subset of samples—the uncensored ones (for which $e_i=1$)—reveals the target variable $T_i$ that we want to estimate; the other censored points (for which $e_i=0$) only present a lower bound on the unobserved $T_i$. We note that the reviewer’s comment is aligned with the motivation behind our flagship method—fused calibration—that offers a rigorous strategy to seize the most informative censored points in addition to the uncensored ones to further enhance statistical efficiency. Indeed, our experiments show that our proposals—especially the fused method—attain better performance than the naive approach despite utilizing less data.

Extreme censoring rates

The theoretical bounds provided in Section 3.3 are valid regardless of the censoring rate. Please refer to GLOBAL RESPONSE: EXPERIMENTS (Robustness analysis to extreme censorship rates) for an experiment that validates our argument.

Marginal vs. PAC-type guarantee

The key difference between a marginal LPB and a PAC-type LPB lies in whether one is interested in obtaining valid coverage conditional on the calibration set (PAC) or not (marginal). In other words, the advantage of a PAC-type guarantee is that it characterizes how the randomness in the choice of the calibration set affects the validity: this is reflected by the “$\delta$ term” in Theorem 3.1 and Theorem 3.2, which shrinks as the size of the calibration set increases. In our work, we employ a PAC-type guarantee, which aligns with the typical scenario in survival analysis where the user is given a specific calibration set (often of limited size) and aims to obtain valid results conditioned on this set.

Valid lower bound on $\tilde{T}$ leads to a valid lower bound on $T$

We apologize if the sentence in lines 162-163 is confusing. For convenience, we repeat it here: “Notably, this $\hat{L}(X; {\hat{\tau}_{\text{naive}}})$ is a PAC-type LPB on $\tilde{T}$, and thus also a PAC-type LPB on $T$.” This statement relies on the fact that $\tilde{T} \leq T$ by definition, and so in an event where $\hat{L}$ is a valid lower bound on $\tilde{T}$, it will also be a lower bound on $T$. This is the core argument as to why the naive method is valid. We’ll include this clarification in the text. Thank you again for helping us improve our writing.

Readability of Section 3.2

We are uncertain about what the reviewer specifically refers to as “results” in Section 3.2. However, we will take this opportunity to clarify the underlying idea of the fused algorithm presented in that section. The motivating question behind the fused method is this: how can we select informative censored points in addition to the uncensored ones to gain statistical efficiency? This is in contrast with the focused method that only uses uncensored points for calibration. The mechanism through which we choose censored points to participate in the calibration process is the censored selection estimator $\hat s_\tau(x)$, motivated by Proposition 3.1 (Eq. 8). In turn, the fused calibration set should ideally include the censored points that violate Eq. 8 and the uncensored points. To this end, we define the selection rule $\zeta_\tau(X_i, e_i)$ as an “or” operator between the event indicator $e_i=1$ and the censored selection estimator $\hat s_\tau(x)=1$. We acknowledge that the definition of the selection rule $\zeta_\tau(X_i, e_i)$ in Section 3.2 was not clearly motivated, which may have led to confusion. We will implement the above discussion to the text to enhance clarity. We greatly appreciate your feedback!

Questions

Extreme weights

That’s an excellent question! Handling extreme weights is of great interest, and fits into the broader context of weighted conformal prediction and other statistical methods that leverage weights to address covariate or label shift (Tibshirani et al. (2019); Candes et al. (2023); Gui et al. (2024)). We anticipate facing such an issue when we observe an uncensored sample with $e=1$ although its estimated $\hat{P}[e=1|X=x]$ is extremely small. In our experiments, we haven’t encountered such an issue, but one approach to handling extreme $\hat{w}$ could be bounding the estimated probability $\hat{P}[e=1|X=x]$, as suggested by Candes et al. (2023). From a theoretical perspective, when the true probability $P[e=1|X=x]$ is very low, there is a trade-off between limiting the estimated weights $\hat{w}(x)$ and maintaining accurate weight estimation. This balance directly impacts the coverage guarantee established in Theorem 3.1. According to the theorem, under-coverage is affected by two key factors: (1) a high value of $\hat{\gamma}$, which bounds $\hat{w}(x)$; and (2) the discrepancy between $w(x)$ and its estimator $\hat{w}(x)$. This analysis sheds light on the impact of bounding the weights. Imposing strict bounds on $\hat{w}(x)$ would reduce $\hat{\gamma}$ but would also negatively impact the accuracy of $\hat{w}(x)$. We will include this discussion in the revised manuscript.

The tightness of the LPBs

Intuitively, when the miscoverage estimator can fully utilize the conditional independence assumption, $T \perp C | X$, we expect to obtain tight coverage. This happens for the method developed by Gui et al. (2024), as they showed how to cast the problem as a calibration under covariate shift. Yet, this formulation requires access to $C_i$ for all observations $i=1,...,n$. In the general right-censored setting, it is impossible to fully utilize the conditional independence assumption for coverage estimation: we do not know the value of $C$ of all samples! Therefore, it is impossible to cast the problem as calibration under covariate shift anymore. Yet, we have shown that we can translate the problem to a “calibration under conservative covariate shift”: kindly refer to the inequality in Eq. 5 that builds on Lemma B.1. Hence, unless the transition in Eq. 5 holds in equality, we expect to obtain conservative coverage. This is a natural consequence of having less information. Following the reviewer’s comment, we seek to analytically quantify the extent to which our calibration method is conservative. We believe this can be done by leveraging the set of steps in the proof of Lemma B.1. We hope to include such a result in the revised manuscript.

In what situations would the LPB of the focused and fused calibrations underperform compared to the naive approach?

The focused method can underperform the naive method in settings where the early censorship rate is low, as rigorously stated in Proposition 3.1. The fused method is expected to match or exceed the performance of both the naive and focused methods, as long as the selection rule $\hat{s}(x)$ is well estimated. This is because the goal of $\hat{s}(x)$ is to detect the censored observations that violate the condition in Proposition 3.1. In turn, the fused approach flexibly includes or excludes censored calibration points in a manner that yields more informative LPBs regardless of whether early censoring is rare or common, for example. This flexibility is demonstrated in Figure 5 in “rebuttel_experiments.pdf” attached to the supplementary material. Lastly, we anticipate the fused and naive methods to perform the same when the condition in Proposition 3.1 is violated for the vast majority of samples.

The estimator $\hat s_{\tau}(x)$

As stated above, the goal of $\hat s_{\tau}(x)$ in the fused method is to flexibly include or exclude censored calibration points in a manner that yields more informative LPBs. It’s important to note that even a poorly estimated $\hat s_{\tau}(x)$ ensures valid coverage. This is because the weights of the fused method are defined as $w(x)=1/P[e=1 | X=x]$ if $\hat{s}(x)=0$ and $1$ otherwise. Notice that this condition does not involve the oracle selection rule $s_\tau(x)$, but solely its estimate $\hat s_\tau(x)$. Thus, the consequence of relying on a poor estimation $\hat{s}(x)$ is reduced efficiency, not invalid coverage. We will clarify these points in the revised text.

References

Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal prediction under covariate shift. NeurIPS, 2019.

Emmanuel Candes, Lihua Lei, and Zhimei Ren. Conformalized survival analysis. JRSS Series B: Statistical Methodology, 85(1):24–45, 2023.

Yu Gui, Rohan Hore, Zhimei Ren, and Rina Foygel Barber. Conformalized survival analysis with adaptive cut-offs. Biometrika, 111(2):459–477, 2024.

Comparison to other methods

Initially, we did not compare our methods to others as no existing techniques provide finite-sample coverage guarantees in the general right-censoring setup. We agree with the reviewer that such a comparison can shed light on the importance of our contribution, especially if the uncalibrated model or an asymptotic calibration method would not obtain the required coverage in practice.
In response, we now include comparisons to the uncalibrated model and to the method developed by Qi et al. (2024), namely the CSD method. We kindly refer the reviewer to our GLOBAL RESPONSE: EXPERIMENTS, showing that for some synthetic datasets, the uncalibrated model and asymptotic CSD produce LPBs that undercover the test time-to-event variables. This is in contrast with our methods, which always attain valid coverage.

Analysis of synthetic experiments from the initial submission and real data experiment

Thank you for your thoughtful analysis of our experiments. Following the reviewer's comment, we discuss below the results of Settings 4 and 6 from the initial synthetic datasets and the TCGA dataset. Setting 4: As pointed out by the reviewer, the coverage rate of the fused method is similar to that of the naive approach. However, notice that the LPBs generated by the fused method are higher than those of the focused and naive methods. Setting 6: Here, all methods produce valid LPBs with similar values. Although the overlapping whiskers make it difficult to draw definitive conclusions about performance, we attribute the slight underperformance of the fused method to inaccuracies in estimating the ideal selection rule $s(x)$. TCGA dataset: The fused and focused methods yield LPBs higher than the naive approach by a small but noticeable margin. While the improvement may appear modest, providing reliable and more informative predictions with LPBs that are a few months higher can have a meaningful impact on decision-making for breast cancer patients. We will include a discussion along these lines in the text to improve the interpretability of our experiments, thank you for raising this point.

Experiments reflecting a general right-censoring setting

We respectfully disagree with the reviewer and apologize for the confusion.

To be sure, all our experiments follow the general right-censoring setting.

The right-censoring framework refers to scenarios where, for each sample, we observe either the survival time $T$ or the censoring time $C$, but not both. In all experiments, the observed data consists of the feature vector $X$, the variable $\tilde{T} = \min(T, C)$, and an indicator variable $e$, specifying whether $\tilde{T}=T$ or $\tilde{T}=C$. We never observe both $T$ and $C$.

Confusing notations

Indeed, $q(X)$ is deterministic given $X=x$. We will clarify this point in the revised manuscript. Regarding the formula $E[P(\tilde{T}<q_{\tau}(X)|X=x,e=1)]$, you are correct, there is a typo. The correct form is $E[P(\tilde{T}<q_{\tau}(X)|X,e=1)]$, as the expectation is taken over draws of $X$. Thank you so much for catching this typo! We will correct it in the revised version.

References in arXiv format

Thank you for this comment, when appropriate, we will revise and correct the format of the references.

Report average LPB

Our figures include box plots showing the distribution of the LPBs and their median. In the synthetic data, we presented the relative LPB to improve readability (as otherwise, the y-axis would have a different scale for each setting). Following this comment, we will also include the average LPB values. Please refer to Tables 3 and 4 in “rebuttal_experients.pdf” of the Supplementary Material.

Coverage rate in the real data experiment

It is impossible to compute the coverage rate for real survival data because $T$ is unknown for censored subjects—this is precisely what makes the right-censoring setting complex and widely studied.

Additional real datasets

We kindly refer the reviewer to GLOBAL RESPONSE: EXPERIMENTS (real data).

References

Shi-Ang Qi, Yakun Yu, and Russell Greiner. Conformalized Survival Distributions: A Generic Post-Process to Increase Calibration. ICML, 2024.

Thank you for your feedback! We are glad that the reviewer appreciates the importance of our contribution and also for confirming the correctness of our guarantees. 

My primary concern is that both the focused and fused calibrations are based solely on a subset of the data, which may lead to wasted information.

We must emphasize that the whole point of conformal survival analysis methods is this: by solely relying on a subset of the data, one can generate more informative LPBs while maintaining validity. Finding such a subset is the cornerstone and key novelty of Candes et al. (2023) and Gui et al. (2024). This is also our key contribution, but in contrast to prior work, we are the first to handle general right-censored data with finite sample guarantees.

Sensitivity to high censoring rates

When the censoring rate is extremely high (say 95%), we mostly have $\tilde{T} = C$. In this setting, the entire survival analysis task becomes intrinsically difficult. It would be hard to estimate $T|X$, as we would barely observe the time-to-event variable $T$. That is, regardless of the calibration step, it would be hard to obtain a useful estimate of the LPB. Of course, it would also be hard to estimate the weights $w(x)$ and the selection rule $s(x)$ for our methods.

We agree that in such extreme cases, it would be advisable to apply the naive method to ensure validity. We will include this discussion in the revised paper.

Yet, the above extreme setup is arguably not a common use case, as reflected by our real-world experiments: the censoring rate varies between 36.6% and 86.6% for the 6 real-world datasets we explored, and the fused method consistently performed the best among the valid calibration methods. Notably, the fused method rigorously tackles the concern raised by the reviewer regarding the utilization of the data, as we detail below. 

I doubt how often the condition in Proposition 3.1 (Eq. 8) is satisfied

This condition rigorously reveals when the focused method would outperform the naive approach. Importantly, this condition depends on the data, i.e., it is impossible to state apriori that one method would always be better than the other. This is precisely the key challenge addressed by the fused method, which enjoys the benefits of both the naive and focused methods. 

Our experiments support this argument. Specifically,

(1) In synthetic settings 1, 3, 4, and 6, as well as real-world datasets TCGA, churn, and NACD, we can see that the focused method outperforms the naive approach. This indicates that the condition in Eq. 8 tends to hold.

(2) In synthetic setting 5, as well as real-world datasets SUPPORT, METABRIC, and GBSG, it is evident that the naive approach outperforms the focused method. This indicates that the condition in Eq. 8 tends to be violated. 

(3) In all the synthetic and real-world experiments, the fused method performs as well as the best among the naive or the focused methods. Further, in synthetic settings 2, 3, and 6, the fused outperforms both the naive and focused methods. This highlights the importance of carefully selecting the subset of calibration points.

The effect of poor estimation of $s(x)$ on statistical efficiency

We agree. A highly inaccurate estimation of $s(x)$ would prevent us from fully exploiting the potential of the fused calibration method. A rigorous analysis of the effect of inaccurate estimation of $s(x)$ is highly challenging not only because it involves an analysis of the training process, but also because this analysis depends on the distribution of the data. We hope to conduct such an analysis in future work, but we must stress that this is very challenging.

Meanwhile, we emphasize that in our experiments we fit a random forest model with the same hyperparameters for all datasets (both real and synthetic). Despite this simple design choice, we observe a very nice gain of the fused method compared to the naive and focused methods. Specifically,

(1) The GLOBAL RESPONSE: EXPERIMENTS (Robustness analysis to extreme censorship rates) demonstrates that the fused method provides a statistical gain compared to both the naive and focused methods over a wide spectrum of censoring rates, between 20% and 80%.

(2) The ablation study in Section D of the supplementary “rebuttal_experiments.pdf”, demonstrates the robustness of our approach to the choice of the random forest’s hyperparameters.

Thank you once again for your insightful comments. We absolutely agree with the reviewer that it is important to discuss the limitations of our methods. We will expand the current discussion in the manuscript regarding the estimation of the weights: we will include a discussion on the effect of extreme weights, strategies to mitigate this problem, and also emphasize that it is preferred to use the naive method in cases where the censorship rate is extremely high. We will also explain that a poor estimation of $s(x)$ can deteriorate statistical efficiency.

We very much appreciate your positive feedback and interest in our work. In what follows, we address your comments in detail.

Comparable performance in Setting 6 for all methods

Interesting point! Recall that in Section 3.1, we formulate the condition for which the naive method will outperform the focused approach, which gives rise to our proposal to enhance efficiency by estimating the ideal selection rule function $s(x)$. Importantly, if the estimate $\hat{s}(x)$ of $s(x)$ is accurate, our fused method should enjoy the benefits of both the focused and naive approaches. In the synthetic data experiment, the fused method generates the most informative LPBs for all settings, except for Setting 6, where the performance of the fused and focused are comparable to that of the naive approach; perhaps with a slight gain for the naive (it’s hard to make a sharp conclusion as the whiskers of all methods are overlapping). In light of the above explanation, we explain this discrepancy by the inaccurate estimation $\hat{s}(x)$ of the ideal selection rule $s(x)$. We will add this discussion to the revised manuscript.

Upper predictive bounds in survival analysis

Thank you for raising this insightful question. Researchers often prioritize the lower predictive bound for survival time because it tends to provide more actionable insights in practice. For instance, knowing a minimum guaranteed survival time is particularly useful in clinical decision-making. Exploring the upper predictive bound is an excellent research direction with potential applications, such as in treatment optimization and automatic assignment. However, this approach faces significant challenges. One practical issue is that in datasets with substantial censorship, higher quantile estimators often lack sufficient data to train effectively. For example, estimating the 95th percentile survival time in a dataset with 10% censorship is inherently difficult. As a result, the calibration process might yield uninformative, overly large upper bounds. To our knowledge, there has been limited work explicitly addressing upper predictive bounds in survival analysis until very recently. The method introduced by Holmes et al. (2024) generates upper bounds with finite-sample guarantees, though it was published after our work. We agree this is a promising area and will discuss the challenges and opportunities for two-sided bounds in the final version of the paper.

References

Chris Holmes, and Ariane Marandon. Two-sided conformalized survival analysis. arXiv:2410.24136, 2024.

LPB is a degenerated output compared to the entire distribution

Our approach aligns with the work of Candes et al. (2023) and Gui et al. (2024), among others, which focus on LPB estimation. However, we recognize that there are scenarios where distributional estimation may be preferred over LPB estimation alone. In such cases, we can utilize our method to calibrate multiple quantiles in a manner similar to the CSD approach by Qi et al. (2024), achieving a PAC-type LPB estimation of the entire survival distribution. In fact, we implemented this approach for measuring our methods’ calibration metrics in Figures 2 and 4 in the supplementary “rebuttal_experiments.pdf.”

Sensitivity to high censoring rates: assess the method's performance across varying levels of censoring

The theoretical bounds provided in Section 3.3 are valid regardless of the censoring rate. Please refer to GLOBAL RESPONSE: EXPERIMENTS (Robustness analysis to extreme censorship rates) for an experiment that validates our argument.

Include censoring rates in both synthetic and real datasets

Thank you for this constructive comment. Please refer to Tables 2 and 5 in the supplementary “rebuttel_experiments.pdf,” summarizing the censorship rates for the datasets used in the new experiments. Notably, both the synthetic and real-world datasets cover a broad range of censorship rates, ensuring comprehensive evaluation.

Double robustness in practice

Great comment! Providing an accurate characterization of the effect of inaccurate estimates on the coverage rate could be an exciting future direction to explore. Our theoretical analysis in Section 3.3 provides an initial step towards this ambitious goal. We remark that Theorem 3.1 and Theorem 3.2 require that either $q$ is well estimated or $w$ is well estimated. The exploration of potential strategies to mitigate the impact of inaccurate estimation of the weights is intriguing. For example, one approach to improve the stability to inaccurate estimations is to avoid extreme weights. This can be done by bounding/clipping the estimated weights, as suggested by Candes et al. (2023). In the interest of space, we kindly refer the reviewer to our response to Reviewer bciN (extreme weights) for further discussion on this topic.

Questions

Why does an inaccurate $\hat{s}_{\tau}$ not affect the coverage guarantee

We thank the reviewer for allowing us to clarify this point. Recall that the weights of the fused method are defined as follows: $w(x)=1/P[e=1 | X=x]$ if $\hat{s}(x)=0$ and $1$ otherwise. Notice that this condition does not involve the oracle selection rule $s(x)$ !! Instead, it relies solely on its estimate $\hat{s}(x)$. As a result, even a poorly estimated $\hat{s}(x)$ of the oracle $s(x)$ ensures valid coverage. The intuition here is that the goal of $s(x)$ is to enhance statistical efficiency, and so the consequence of relying on its poor estimate is reduced efficiency, not invalid coverage. We will clarify this point in the text.

References

Candes et al. Conformalized survival analysis. JRSS Series B: Statistical Methodology, 85(1):24–45, 2023.

Gui et al. Conformalized survival analysis with adaptive cut-offs. Biometrika, 111(2):459–477, 2024.

Qi et al. Conformalized Survival Distributions: A Generic Post-Process to Increase Calibration. ICLR, 2024.

Thank you for your positive feedback! We appreciate your constructive, detailed, and valuable suggestions. In what follows, we address your concerns in detail.

Weaknesses

Simplistic synthetic datasets

Thank you for this great suggestion! We revised our synthetic experiments and generated more realistic and challenging data, with a complex conditional distribution $(T, C) | X$, early censoring, early time-to-event occurrences, and end-of-trial censorship. We believe these better mimic the complexities encountered in practical use cases. Please refer to GLOBAL RESPONSE: EXPERIMENTS for more details and results.

Comparison with other methods

Initially, we did not compare our methods to others as no existing techniques provide finite-sample coverage guarantees in the general right-censoring setup. We agree with the reviewer that such a comparison can shed light on the importance of our contribution, especially if the uncalibrated model or an asymptotic calibration method would not obtain the required coverage in practice.
In response, we now include comparisons to the uncalibrated model and the CSD method by Qi et al. (2024). We did not compare with the asymptotic methods by Qin et al. (2024) and by Meixide et al. (2024) as neither provides software implementations. Additionally, these contributions are concurrent with ours according to ICLR guidelines. We kindly refer the reviewer to our GLOBAL RESPONSE: EXPERIMENTS, showing that for some synthetic datasets, the uncalibrated model and asymptotic CSD produce LPBs that undercover the test time-to-event variables. This stands in contrast with our methods, which always attain valid coverage. Lastly, we note that our focus on finite-sample guarantees is motivated by applications where small sample sizes are common. This is exemplified by the real-world breast cancer TCGA dataset used in our experiments. To clarify, we are not arguing that methods without finite-sample guarantees will fail in such cases, and we acknowledge the broad applicability and great utility of asymptotic approaches. However, we view finite-sample validity as an additional safeguard. We believe our numerical experiments support this perspective, demonstrating that the asymptotic method can fail to achieve valid coverage in certain settings. We thank the reviewer for this valuable comment, which we believe improves our paper.

Evaluation on one real-world dataset is insufficient

We refer the reviewer to the GLOBAL RESPONSE: EXPERIMENTS for new real-world data experiments, comparison to baseline methods, calibration metrics, and more.

Evaluating distribution calibration metrics

We very much appreciate your suggestion. We now include the following metrics in our experiments: c-index, integrated brier score (IBS), and distribution calibration (D-Cal). The results for the synthetic datasets are summarized in Figure 2 in the supplementary “rebuttal_experiments.pdf”:

• c-index: our calibration methods do not affect the c-index in practice. Further, all methods demonstrate the same performance under this measure.

• IBS and D-Cal: we agree that IBS and D-Cal are valuable metrics for evaluating the extent to which a survival analysis model is calibrated. However, these measures can penalize methods that generate valid but conservative LPBs. Indeed, following Figure 2 in the supplementary PDF, one can see that the baseline methods (uncalibrated model and CSD) outperform our methods in terms of IBS and D-Cal, however the baselines do not achieve the correct coverage level—see, e.g., settings 1,3,4. This discussion underscores that the IBS and D-Cal metrics are not perfectly aligned with our goal of evaluating whether a survival analysis model provides valid LPBs with a coverage rate higher than the desired level. In fact, these metrics might suggest poor performance for methods that actually provide valid but conservative LPBs. Following the above, we believe that the development of a method that performs well under these calibration metrics while attaining valid LPBs is a promising future direction.

Unfortunately, we cannot analyze the correlation between coverage and IBS/D-Cal for real-world datasets, as it’s impossible to compute the coverage. However, as shown in Figure 4 of the supplementary “rebuttal_experiments.pdf,” a similar trend emerges: e.g., the naive method performs worse in terms of IBS and D-Cal compared to the uncalibrated model. This occurs although we have reasons to believe that the naive method yields valid but conservative LPBs. Lastly, we point out that in terms of c-Index, all the methods perform the same on real-world datasets.

References

Qi et al. Conformalized Survival Distributions: A Generic Post-Process to Increase Calibration. ICLR, 2024.

Qin et al. Conformal predictive intervals in survival analysis: a re-sampling approach. arXiv:2408.06539, 2024.

Meixide et al. Uncertainty quantification for intervals. arXiv:2408.16381, 2024.

We appreciate your constructive feedback and suggestions. In what follows, we address your concerns in detail.

The novelty of the proposed methods

See GLOBAL RESPONSE: CONTRIBUTIONS.

Comparison to other methods

See GLOBAL RESPONSE: EXPERIMENTS.

Effect of inaccurate weights

When relying on inaccurate weights, one might obtain either under-coverage or over-coverage in practice. Our experience suggests that our proposed methods are fairly robust to inaccurate weights. This observation is in line with related conformal methods that theoretically require accurate weights for valid coverage, but they often achieve the desired level in practice (Tibshirani et al. (2019); Candes et al. (2023); and Gui et al. (2024)).

From a theoretical perspective, Theorem 3.1 establishes a lower bound on the attained coverage rate based on the magnitude of the estimation error of the weights. This quantifies the worst-case drop in coverage stemming from inaccurate weight estimation.

Evaluating the accuracy of the weights, reducing sensitivity to errors

The reviewer’s comment is of great interest, and it touches upon the broader context of weighted conformal prediction and other statistical methods that leverage weights to address covariate or label shift (Tibshirani et al. (2019); Candes et al. (2023); Gui et al. (2024)).

Unfortunately, assessing the accuracy of the estimated weights is challenging, as it is equivalent to estimating the ground truth weights $w(x) = 1/P[e=1 \mid X]$, which is generally infeasible.

As for strategies to reduce the sensitivity to estimation errors, one approach to improve stability is to avoid extreme weights. This can be done by bounding/clipping the estimated weights, as suggested by Candes et al. (2023). In the interest of space, we kindly refer the reviewer to our response to Reviewer bciN (extreme weights) for further discussion on this topic.

The computational complexity of the proposed methods

All methods, including ours, require training a base predictive model, $\hat{q}$. The focused and fused calibration methods require fitting a classifier for weight estimation. The fused calibration method requires training an additional classifier $s(x)$ for each threshold considered. Training these additional classifiers can be done in parallel, and the computational complexity is comparable to that of fitting the base predictive model.

As for the complexity of the calibration process, the naive, focused, and fused calibration schemes involve a linear pass over the calibration samples to estimate the miscoverage rate for a given threshold. This calibration process should only be applied once (offline), and it is computationally negligible in comparison to training the survival analysis model, for example. The inference time is identical to running the base predictive model, as it amounts to invoking the model with a calibrated quantile level. We will further elaborate and clarify this point in Appendix D.2.

Weights and conditional quantile estimation: the effect of hyperparameters

In our experiments, we use a single set of hyperparameters, as detailed in Appendix D, and we apply early stopping on a validation set to avoid overfitting. Despite using such a simple strategy, our methods always attained valid coverage. In response to the reviewer’s comment, we conducted an ablation study to analyze the impact of key hyperparameters on model performance, specifically in estimating weights and conditional quantiles, with a focus on their influence on coverage and the resulting LPBs.

For weight estimation, we test the performance of our methods using either a shallow random forest classifier (max depth 2) or a deeper one (max depth 6). For conditional quantile estimation, we test the methods using either a shallow network (1 hidden layer) or a deeper network (3 hidden layers). The results are presented in Figure 6 in the “rebuttel_experiments.pdf.” Following that figure, we can see that our calibration methods always attain valid coverage, highlighting the robustness of our methods to different design choices of the underlying predictive models.

Theoretical bounds for coverage under inaccurate estimations

The impact of inaccuracies in weight and conditional quantile estimation is characterized by the error terms of Theorems 3.1 and 3.2. Theorem 3.1 shows that the coverage rate is affected by the ratio $\hat w_\tau (x) / w_\tau (x)$ , quantifying how well the weights are estimated. Theorem 3.2 shows that conditional coverage is affected by the difference $\hat q_{\tau(\beta)}(x) - q_{\tau(\beta)}(x)$, quantifying how well the conditional quantiles are estimated.

References

Tibshirani et al. Conformal prediction under covariate shift. NeurIPS 2019.

Candes et al. Conformalized survival analysis. JRSS Series B: Statistical Methodology, 85(1):24–45, 2023.

Gui et al. Conformalized survival analysis with adaptive cut-offs. Biometrika, 111(2):459–477, 2024.