Uncertainty-Calibrated Prediction of Randomly-Timed Biomarker Trajectories with Conformal Bands

We thank the reviewer for their extensive and constructive feedback. We are very glad that the reviewer found the application to be well-motivated, the methodology and experimental validation sound, and the writing clear and accessible. We also appreciate your positive comments on the broad relevance of predicting biomarkers under randomly timed observations, as well as our carefully designed group-conditional covariates, which you noted as among the most comprehensive seen in applied conformal prediction and go beyond what is typically explored in the literature.

In summary, our response addresses the following key points:

1. We address all the suggestions (include the CIs in Table 1 and Table 4 of the paper)
2. We elaborate on the novelty of our work as well as the impact on the NeuRIPS community.

---

Weaknesses

• We agree with the reviewer that the normalizing function $\sigma$ does not depend only on $\hat{Y_t}^{(i)}$. A more complete notation would be $\sigma(\hat{Y_t}^{i},X_t;f)$, with $f$ denoting the predictor and $X_t$ the input at time $t$. This emphasizes that the uncertainty estimate depends not only on the predicted value $\hat{Y_t}^{(i)}$ but also on the covariates as well as properties of the predictor $f$ used to generate it. In the manuscript, we used the simplified notation $\sigma(\hat{Y}_{t}^{(i)})$ for readability. We will clarify in the final version that this is a simplified form and that the normalizing function is indeed dependent on other quantities.

  To make this dependence concrete:

  • For Gaussian Process-based predictors (e.g., DKGP, DME), $\sigma$ returns the posterior predictive standard deviation from the model.
  • For frequentist ensemble methods (e.g., Bootstrap, Monte Carlo Dropout), $\sigma$ corresponds to the empirical standard deviation across multiple forward passes or resampled models at time $t$ .

• Thank you for the helpful comment. We will make the clarification in Problem 3.1 as you suggested.

• Regarding your question about the CIs, in the revised manuscript we have included the CIs in both tables. Here we present for reference the updated results for the Table 1 of the main manuscript:

Table: Youden-optimized discrimination on $z$-standardized rate of change (RoC) and lower-bound rate of change (LRoC) for MCI converters, with 95% bootstrap confidence intervals.

We have also updated the Table 4 of Appendix accordingly.

• Also we plan to release the code upon acceptance and we have made the modifications in the checklist.

Questions

We appreciate the opportunity to clarify the broader significance of our work. To the best of our knowledge, this is the first conformal prediction framework tailored to randomly timed biomarker trajectories, a setting that arises in clinical studies. Existing CP methods assume fixed or synchronized timepoints across subjects, which makes them incompatible with the asynchronous and irregular observation patterns present in real-world clinical data. While our nonconformity score is a key adaptation, our contribution goes beyond that: we formally extend split conformal prediction to the setting where time index $\mathcal{T}$ is random. This includes a normalization mechanism that enables inference at arbitrary, unseen timepoints and supports guarantees even under irregular follow-up—a critical requirement for clinical deployment with real data from clinical studies. In Appendix F, we explain in detail how methods designed for fixed-time trajectories cannot be applied in our setting, which highlights the importance of our results mainly from a practical (rather than a technical) perspective. This problem setting is of particular interest to the NeurIPS AI for Health and Medicine community. Our approach not only enables distribution-free guarantees in this setting but also highlights real-world challenges (e.g., misaligned visit times, subgroup disparities, calibration under irregular sampling) that are underexplored in the CP literature. Additionally, our paper outlines several methodological and applied directions for future research—such as multivariate conformal prediction, adaptive methods for evolving trajectories, and group-conditional guarantees for intersectional subgroups—which we believe will help bridge the gap between theory and practice in clinical ML applications.

Sorry for the misunderstanding. While it is true that our approach builds on the general split conformal prediction framework, our novel nonconformity score is essential for obtaining conformal bands for randomly-timed trajectories. In the following, we clarify why standard CP methods cannot be applied to obtain valid simultaneous coverage guarantees at multiple random time points, providing a more detailed explanation of Appendix F from the manuscript.

The standard approach for obtaining simultaneous coverage over multiple time points would be to: i) apply split conformal prediction for each time point for which observations are available in the given dataset, and ii) employ a union bound argument to obtain trajectory-level coverage guarantees. However, this standard strategy cannot be applied in our setting, where each subject has a different number of observations occurring at different time points. More specifically, there is no common set of time points across subjects over which a union bound could be applied. To adapt this naive approach to our setting, one could select a subset of the data corresponding to a fixed set of time points (e.g., 3, 7, 9, and 12 months) and extract only the corresponding observations from each subject. However, this leads to substantial dataset reduction. In a simple experiment following this approach, we found that only 37 trajectories with observations at all four selected time points remained out of the total dataset of 2200 trajectories. Beyond dataset reduction, this method does not allow obtaining valid conformal bands for any test subject. In particular, since this standard approach provides conformal intervals only at the pre-specified time points $t$ (e.g., 3, 7, 9, 12 months), it cannot be used to provide valid confidence intervals for, e.g., a test subject who visits the clinic at months 4, 5, and 14. Our normalized nonconformity score resolves that issue by incorporating the errors for all random time points into a single nonconformity score, which can be directly used to design conformal bands for any test subject. Thus, we provide conformal bands with trajectory-level coverage guarantees without the need for union bounding—which is not applicable to our setting as explained above.

Thus, our nonconformity score::

• Enables valid inference at unseen and asynchronously sampled time points;
• Provides trajectory-level coverage without the need for a union bound.

We thank the reviewer for their feedback and their thoughtful review. We are happy that the reviewer found our problem—reconstructing biomarker trajectories from irregular time points and quantifying uncertainty through prediction intervals—genuinely useful for clinical and other application areas, and saw the potential benefit of our approach.

In summary, our response addresses the following key points:

1. We explain how our conformal method compares with five baseline uncertainty quantification methods (e.g., Gaussian-process based bands) based on our comparative case from Section 4.
2. We elaborate on how our conformal algorithm can be extended to account for multiple biomarkers.

---

Weaknesses

• Our current comparison in Section 4 consists of a total of five baselines, including Gaussian Process-based baselines—see also Appendix for comparative experiments for Sections 5 and 6. Specifically, both DKGP (Deep Kernel Gaussian Process) and DME (Deep Mixed Effects model) are GP-based methods capable of estimating uncertainty bands. In figure 2 you can notice that we compare the DKGP and DME with their conformalized counterparts, where our approach is able to achieve the nominal coverage with tighter bands.

• While our current implementation focuses on single-biomarker prediction bands, our conformal algorithm can be directly extended to multivariate predictors that jointly model multiple biomarker trajectories. As noted in the future work section, we are actively pursuing this direction. A natural generalization involves defining the nonconformity score as the maximum normalized residual across all biomarkers and timepoints, allowing for multivariate trajectory-wise conformal coverage. More specifically, this means that in the current expression of the nonconformity scores we would add an additional index $j=1,...,B$, ranging over all $B$ biomarkers we care about and the same index would be added in the true and predicted biomarker values, which will now be denoted by $Y_{t,j}^{(i)}$ and $\hat{Y}_{t,j}^{(i)}$, respectively.

Note also that our method is not intended to substitute specific predictive models, but rather to provide a conformalization procedure that can be applied to any base predictor—including multivariate models such as the Gaussian Process disease progression framework proposed by Lorenzi et al. While GP-based models are powerful, they come with limiting assumptions (e.g., Gaussianity, stationarity) that may not hold in clinical data. Our conformal approach complements these models by providing distribution-free coverage guarantees in the presence of model misspecification.

Questions

GP-based methods estimate uncertainty based on modeling assumptions, but often produce overly conservative intervals, as shown in Figure 2 for DKGP and DME. Frequentist methods like Bootstrap and Monte Carlo Dropout can underestimate uncertainty and lead to narrow, miscalibrated bands. Our conformal method improves both cases: it can be applied on top of any of these predictors to produce narrower or better-calibrated intervals that achieve valid coverage guarantees. Unlike model-based or sampling-based methods, our conformal prediction method makes no distributional assumptions and provides formal coverage guarantees that hold by construction.

We would like to thank the reviewer for acknowledging that our paper is well structured and clearly written. In summary, our response addresses the following key points:

1. Major concern: We elaborate on the contributions of our conformal method based on our detailed comparison in Appendix F. We explain that our method: i) is applicable to randomly-timed trajectories via the novel use of a normalizing function, ii) provides provable coverage guarantees in the context of randomly-timed trajectories, and iii) is very impactful for practical applications with randomly-timed trajectories, as approaches for fixed-time trajectories cannot be applied to such settings.
2. We provide updated results that address the issue pointed out by the reviewer regarding performance disparities across groups in population conformal prediction.
3. As requested, we discuss the limitations of our approach, and we have already revised the manuscript accordingly.

In our response below, we aim to address in detail each of these concerns, along with several additional questions related to the experimental results.

---

Weaknesses

• Our core technical contribution is the introduction of a normalizing function that enables conformal prediction for randomly timed trajectories. This adaptation allows nonconformity scores to be evaluated at arbitrary, subject-specific timepoints—supporting valid coverage even under irregular and asynchronous sampling. The normalizing function enables us to compute a normalized prediction error across all time steps, ensuring that no component within the maximum dominates the others in terms of scale. The above are explained in detail in Appendix F, but in the revised manuscript we will provide intuition for the reader earlier in Section 3. Thanks to this methodological advance, our conformal method is the first to provide formal coverage guarantees for the setting of randomly-timed trajectories. Beyond its theoretical contributions, our approach addresses a critical practical gap: it is, to our knowledge, the first to implement conformal prediction in the setting of randomly-timed trajectories. Prior conformal methods (e.g., Stankeviciute 2021, Yu 2023, Cleaveland 2024) assume fixed, synchronized time grids across subjects, making them unsuitable for real-world clinical data where observation schedules vary widely. Our framework removes this constraint, unlocking conformal prediction for a broad range of applications in longitudinal healthcare modeling.

• We thank the reviewer for pointing out the discrepancy in stratified coverage. Upon review, we identified that when we evaluated the population conformal prediction per race-group we had excluded several subjects due to fallacious filtering. We sincerely thank the reviewer for catching this issue. We revised the figure accordingly and below we attach the results for the Race. The rest of the results are unaffected. We add the corrected results too as a table.

Table: Comparison of coverage under population vs. group-conditional conformal prediction, stratified by race (White, Black, Asian)

• Indeed, we do not explicitly discuss limitations of the current approach (only our section on future work hints to open problems). We will include a dedicated paragraph on limitations in the final version of the paper. One key limitation of our current group-conditional approach is its lack of scalability with respect to the number of covariates. While stratifying by a single covariate (e.g., sex, race, diagnosis) allows for interpretable subgroup analysis, it becomes impractical to condition on combinations of multiple covariates as this may lead to very small subgroup sample sizes (e.g., Asian, Homogenous for APOE4). Even though our method can be applied to such subgroups, in practice we expect that small sample sizes corresponding to underrepresented subgroups will yield uninformative conformal bands. This restricts the practical applicability of the analysis and may overlook intersectional disparities in model calibration. As discussed in our future work, we aim to extend the group-conditional procedure to intersectional subgroups (e.g., Black females aged 55–65), which would provide more clinically relevant and equitable coverage guarantees. Another limitation is that our current framework cannot be directly applied to external datasets while preserving the original coverage guarantees, unless a new calibration step is performed on the target study. When applied to external studies with potentially different demographic distributions, data acquisition protocols, or diagnostic criteria, the assumptions for valid CP are violated. Therefore, without prior recalibration, applying our method to unseen cohorts may lead to invalid or misleading prediction intervals.

Questions

• We would like to clarify that our contribution goes beyond simply changing the predictive model. The key methodological novelty lies in the introduction of a time-aware normalizing function used in the nonconformity score, which enables conformal prediction in the context of randomly timed longitudinal trajectories. Existing methods, such as Yu et al. (2023), assume that all subjects share a common set of fixed observation times across the training, calibration, and test sets. These approaches define normalization terms σ_t only at pre-specified time points and therefore provide valid conformal coverage only at those points, limiting their applicability to real-world data where visits occur at variable and often irregular times. Attempting to apply such methods to real biomarker trajectories would necessitate discarding a large portion of the dataset — e.g., in our experiments, selecting four fixed time points (3, 7, 9, 12 months) yielded only 37 usable trajectories out of 2200. To address this, we introduce a normalizing function σ(·) that produces predictive uncertainty estimates σ(Ŷ_t) at any time point t, even if no observations exist at that time in the calibration set. This allows us to compute normalized nonconformity scores |Ŷ_t - Y_t| / σ(Ŷ_t) at arbitrary, subject-specific test-time points, thus supporting valid coverage under asynchronous and irregular sampling schedules. Crucially, this normalization ensures that no individual time point dominates the score due to scale differences, and that prediction intervals adapt to time-varying uncertainty. We explain the full technical details of this construction in Appendix F, but in the revised manuscript, we will provide earlier intuition in Section 3 to help guide the reader through this key innovation.

• The confidence intervals are also visualized in the Figure 2a. This is the table with the 95% CI of the coverage metric across the folds:

Table: 95% confidence intervals for marginal coverage and average interval width across methods, before and after conformalization.

I thank the authors for their response, including corrected experimental results and the detailed explanation of the novel normalizing function $\sigma(\cdot)$ that produces predictive uncertainty estimates $\sigma(\hat{Y}t)$ at any time point $t$ for those time points that are missing. I acknowledge that this is a novelty that I overlooked before, and appreciate the authors' agreement on highlighting this key novelty in their main body for future readers. A quick follow-up question: Did you use $\mathcal{D}_{\text{train}}$ or $\mathcal{D}_{\text{cal}}$ to train the normalizing function $\sigma(\cdot)$?

I’m tentatively considering increasing my score to borderline reject (or potentially higher) in light of the novelty of using a predictor function to estimate the normalizing score, but I would prefer to wait for the authors’ next round of responses before making a final decision. The reason I remain slightly negative is that, in my personal view, the significance of this addition to the broader conformal prediction literature is not sufficient for a recommendation to NeurIPS. Please feel free to correct me or disagree, my understanding of the approach proposed by [Yu et al.] is that it can be implemented even without the normalization term, i.e., let $S_i = \max_{t \in [T]} |\hat{Y}_t^{(i)} - Y_t^{(i)}|$, which corresponds to the maximum residual one could make across the observation window for trajectory $i$. This score (without normalization) preserves the coverage guarantee. The normalization term is more optional and is more for improving adaptivity and efficiency (width) of their resulting conformal prediction bands. Without normalization, the core intuition behind the nonconformity score design appears to be the same between [Yu et al.] and this paper, regardless of the randomness in observing/missing values at particular time steps.

Minor: There is a missing citation on line 761 in Appendix F

We thank the reviewer for their feedback and for highlighting the key strengths of our work. We are delighted that reviewer recognizes that we address a critical limitation of prior methods, while achieving nominal coverage with tight prediction bands compared to baseline approaches. We are also grateful for the reviewer’s appreciation of our empirical demonstration on brain biomarkers and the importance of uncertainty calibration in heterogeneous clinical populations.

In summary, our response addresses the following key points:

1. We explain that our conformal method is designed so that it can directly handle irregular trajectories without the need of temporal alignment.
2. We expand and enhance the comparison between the RoC and LRoC with the requested metrics.

---

Weaknesses

• In our setting, each subject's time axis is defined relative to their baseline visit, and our predictors are trained to forecast future biomarker values based on this subject-specific temporal reference frame. Also, we would like to clarify that our method is specifically designed to handle trajectories that are not temporally aligned across subjects. In our formulation, the set $\mathcal{T}$ of time points at which observations are collected for a given subject is treated as a random variable, and our coverage guarantee in Theorem 3.2 is stated with respect to the joint distribution over the triplet $(X, \mathcal{T}, Y)$. This formulation explicitly incorporates the randomness of observation times without relying on any assumption of a shared or aligned time axis across subjects. Furthermore, our nonconformity scores are computed at subject-specific timepoints relative to each individual's baseline, and we do not perform any form of temporal registration or warping. The validity of our coverage guarantee holds under this irregular and asynchronous observation setting and is rigorously proven in Appendix B.
• We thank the reviewer for raising this point regarding group specification. To clarify, the grouping function $G: \mathbb{R}^d \rightarrow \mathcal{G}$, introduced in line 263, is instantiated using single categorical covariates in our current experiments. As described in Section 5.2, we evaluate group-conditional conformal prediction across five clinically relevant stratifications: sex, race, education level, diagnosis, and APOE4 allele status. These covariates are well associated with Alzheimer’s Disease and Brain Ageing [1,2]. We will elaborate about this in the revised manuscript.
• We thank the reviewer for the insightful suggestion to contrast RoC and LRoC using threshold-free evaluation metrics. In response, we conducted a comprehensive analysis across ten trajectory models (e.g., DKGP, CP-DRMC, Bootstrap), using partial AUC (PrAUC) computed at varying sensitivity thresholds. Rather than relying on full ROC-AUC, which can obscure performance at clinically relevant operating points, we report partial AUC at 0.8 sensitivity, a threshold that offers a strong balance between recall and precision. A sensitivity of 0.8 is clinically appropriate for high-risk screening applications such as early MCI-to-AD conversion detection—prioritizing early identification of converters without overwhelming clinicians with false positives. The difference in partial AUC between RoC and LRoC is negligible for DKGP (Δ = −0.003), reverses in favor of LRoC for DQR (Δ = +0.007), and even improves substantially for DRMC (Δ = +0.032). This supports our claim that LRoC can enhance conservative prediction under uncertainty without meaningfully degrading AUC-based performance. These findings clarify that LRoC does not aim to outperform RoC across all metrics, but rather serves a complementary role—offering robust sensitivity under worst-case assumptions. We have included these comparisons in Appendix Table 4 and updated the manuscript to emphasize the 0.8-sensitivity PrAUC as a clinically justified and empirically stable operating point. We will include the additional metrics in Table 4 of Appendix I.

Questions

• The mean interval width reported in Figure 2 is computed over both indices $i$ and $t$, as briefly mentioned in line 227 of the manuscript. The averaging over $t$ is necessary for a fair empirical evaluation of our method, since the probability distribution in our coverage guarantee is over triplets (X,$\mathcal{T}$,Y), where $\mathcal{T}$ is the set of random points corresponding to a test subject. Our method is specifically designed to handle irregularly sampled trajectories, and the mean interval width we compute provides the empirical average over random subjects with different sets of random time points.

• The coverage differences observed in Figure 4, relative to Figure 2, are primarily due to the use of a common calibration set across the population and group-conditional conformal prediction experiments. Specifically, we used a fixed calibration set comprising 20% of the subjects, which was selected based on the calibration set size exploration for the Population CP (see Appendix G.2, and applied the same set when evaluating Group-Conditional CP. This decision was made to ensure a fair comparison between population and group-conditional methods under consistent data conditions. However, we would not expect the calibration set size that we selected for the Population CP to also provide tight intervals for the Group-Conditional CP. In practice, if we were to apply Group-Conditional CP, we would make another exploration (similar to that in Appendix G.2) for the calibration set size that would result in tight bands for the individual groups. In particular, we would select the calibration set size separately for each group (e.g., CN, MCI, AD) to achieve tighter adherence to the nominal level.

• The prediction horizon is defined individually for each subject as the time span between their baseline and last available observation. On average, this corresponds to 43 months (standard deviation: 32 months), but the horizon varies across subjects due to irregular follow-up schedules. We will clarify this point in the revised manuscript and will also update the notation throughout the paper to make it explicit that both $t_0$ and $t_N$ are subject-specific (i.e., indexed by subject $i$).

  We agree that the mathematical definition of Rate of Change (RoC) in lines 361–364 should be included in the manuscript. In the revision, we will provide an explicit formula for RoC and place it side-by-side with the LRoC definition in Equation (3). Specifically, we define RoC for subject $i$ as:

$$\operatorname{RoC}_i = \frac{\hat{y}_i(t_N^{(i)}) - y_i(t_0^{(i)})}{t_N^{(i)} - t_0^{(i)}}$$

Where:

• $t_0^{(i)}$ and $t_N^{(i)}$ denote the subject-specific baseline and final timepoints,

• $\hat{y}_i(t)$ is the predicted biomarker value at time $t$.

• In our framework, the term “trial eligibility” refers to identifying high-risk subjects likely to experience rapid decline toward Alzheimer’s disease. Such subjects are eligible to be included in a clinical trial and therefore the terms “trial-eligible” and “high-risk” subjects are used interchangeably in the manuscript. In the revised manuscript, we will make sure to clarify the above.

• Table 1 focuses on DRMC as a representative baseline due to space constraints in the main manuscript. However, as shown in Appendix I Table 4, the same conclusion holds across all base predictors—namely, that LRoC improves recall relative to RoC, often with comparable or improved F1 scores. We will include the full set of results in the main paper in the final version where space permits it.

• We will update the corresponding plot to include a decision boundary line with slope equal to the Youden-optimized threshold. This line will clarify how the threshold corresponds to the separation criterion in the LRoC and CP-LRoC.

Limitations

Indeed, we do not explicitly discuss limitations of the current approach (only our section on future work hints to open problems). We will include a dedicated paragraph on limitations in the final version of the paper. One key limitation of our current group-conditional approach is its lack of scalability with respect to the number of covariates. While stratifying by a single covariate (e.g., sex, race, diagnosis) allows for interpretable subgroup analysis, it becomes impractical to condition on combinations of multiple covariates as this may lead to very small subgroup sample sizes (e.g., Asian, Homogenous for APOE4). Even though our method can be applied to such subgroups, in practice we expect that small sample sizes corresponding to underrepresented subgroups will yield uninformative conformal bands. This restricts the practical applicability of the analysis and may overlook intersectional disparities in model calibration. As discussed in our future work, we aim to extend the group-conditional procedure to intersectional subgroups (e.g., Black females aged 55–65), which would provide more clinically relevant and equitable coverage guarantees. Another limitation is that our current framework cannot be directly applied to external datasets while preserving the original coverage guarantees, unless a new calibration step is performed on the target study. When applied to external studies with potentially different demographic distributions, data acquisition protocols, or diagnostic criteria, the assumptions for valid CP are violated. Therefore, without prior recalibration, applying our method to unseen cohorts may lead to invalid or misleading prediction intervals.

[1] Corder, E. H., et al. (1993). Gene dose of apolipoprotein E type 4 allele and the risk of Alzheimer's disease in late onset families

[2] Meng, X., et al. (2012). Education and dementia in the context of the cognitive reserve hypothesis: a systematic review with meta-analyses and qualitative analyses