A Unified Comparative Study with Generalized Conformity Scores for Multi-Output Conformal Regression

Thank you for your feedback and suggestions for improvement.

Missing relation to family-wise error rate (FWER) control methods:

Thank you for raising this point. We do briefly mention multiplicity control approaches in Appendix A (citing Timans et al., 2024 [59]). FWER methods (like Bonferroni or Holm corrections on univariate CPs) are indeed a related direction for constructing multivariate regions, typically by taking Cartesian products of univariate regions $\hat{R}\_i$. Their strength lies in leveraging well-understood univariate CP. However, our work focuses on methods that explicitly model the joint distribution $F\_{Y|X}$ (using e.g., multivariate NFs, DRFs, GMMs, or generative sampling) to construct potentially non-rectangular regions that can better capture output dependencies and often result in smaller region sizes (as seen empirically). While FWER methods are valuable, especially when only marginal predictions are available, the methods we survey and propose aim for potentially tighter regions by directly modeling the multivariate structure. We have revised Appendix A to better clarify this distinction.

Conclusion unclear/short:

We agree that the conclusion was too brief and lacked practical guidance. We have expanded it to provide clear, actionable takeaways based on our unified study and experiments, summarizing the trade-offs:

• If only univariate margin models are available/desired: Use M-CP or CopulaCPTS (fast, but hyperrectangular; may poorly capture dependencies).
• To minimize average region size (potentially sacrificing conditional coverage): Use DR-CP (if density $\hat{f}$ available), else STDQR (if invertible latent model Q available), else PCP (if only samples $\hat{Y}$ available).
• To achieve good conditional coverage and small median size: Use C-HDR (if density $\hat{f}$ available), else L-CP (if invertible Q available), else C-PCP (if only samples $\hat{Y}$ available).
• For good conditional coverage with lowest computational cost among ACC methods: Use L-CP (if invertible Q available).

Questions

1. Relation to FWER: Please see our response above regarding the distinction (modeling joint vs. combining univariate CPs, region shapes, and dependency handling). To better understand the prediction regions produced by FWER control methods, we also provide a qualitative and quantitative comparison with Bonferroni correction on our dataset from Figure 1 in this linked PDF. The method is fast computationally but produces larger regions due to the rectangular shape, similarly to M-CP.

2. Need high-dimensional/unstructured data experiments: We agree on the importance of high-dimensional settings. To this end, we included an experiment on CIFAR-10 image generation (Appendix I), where the output is a 3x32x32 image ($d$=3072). We used a conditional Glow model [34, 54]. The results (Table 3) largely confirm our findings from tabular data: C-PCP, L-CP, and C-HDR demonstrate superior conditional coverage (WSC, CEC) compared to other methods even in this high-dimensional setting. We agree this experiment deserves more visibility and will ensure it is also mentioned in the introduction.

3. How is the density $f(y|x)$ known? Crucially, we do not assume the true density $f_{Y|X}$ is known. All density-based methods (DR-CP, C-HDR, HD-PCP) rely on an estimate $\hat{f}(y|x)$ learned from data. In our experiments, $\hat{f}$ is provided by models like normalizing flows (MQF², Glow), DRF+KDE, or GMMs (detailed in Appendix F.2). Conformal prediction takes this potentially imperfect estimate $\hat{f}$ and calibrates the resulting prediction regions (using the calibration set) to provide valid finite-sample marginal coverage guarantees, regardless of how accurate $\hat{f}$ is. Methods like C-HDR, L-CP, and C-PCP additionally provide asymptotic conditional coverage guarantees as long as the estimator $\hat{f}$ (or sampler $\hat{Y}$, or latent map $\hat{Q}$) converges to the true data generating process.

Thank you for your feedback and recognizing the value of our work.

Proposed scores seem not to have finite-sample conditional guarantees:

You are correct. Our focus is on developing flexible conformal methods applicable to complex, modern generative models (density, sample, or latent-based) with minimal assumptions on the underlying data distribution or the base predictor's accuracy. Under such weak, distribution-free assumptions, achieving exact finite-sample conditional coverage is generally impossible, as proven by Barber et al. (2019) [5]. Instead, we provide asymptotic conditional coverage (ACC) guarantees (Appendix E.2) for C-PCP, L-CP, and C-HDR. This means that as the base model converges to the true distribution and the calibration set size increases, these methods provably achieve the desired conditional coverage level. Our experiments (Fig 3) further empirically validate that these methods also achieve superior empirical conditional coverage in finite samples compared to methods lacking ACC guarantees.

Thank you for the thorough, positive review and your support for acceptance. We appreciate you finding the paper enjoyable and the methods sound. We will correct the reference issues you kindly pointed out in the final version.

Thank you for your valuable feedback.

The paper mentions that CP²-PCP is similar to C-PCP. What is the reason for not including it?

CP²-PCP was proposed concurrently with our submission period and published very recently (ICLR 2025). Hence, we were unable to include it in our empirical comparisons. However, recognizing its relevance, we have added an in-depth discussion in Appendix H comparing CP²-PCP with our CDF-based scores (including C-PCP) and highlighting the similarities and differences in their approaches to achieving conditional validity.

The novelty seems limited to the proposed conformity scores C-PCP and L-CP

In addition to the proposed conformity scores C-PCP and L-CP, this work is the first (to our knowledge) to categorize systematically, implement (in a unified codebase for fairness), and empirically compare a broad range of multi-output conformal methods (marginal, density, sample, latent-based) within a single framework. This synthesis reveals crucial trade-offs (Table 1), motivates our proposed methods (C-PCP/L-CP each address specific gaps), and establishes theoretical connections, which were previously unexplored.

Unclear if/when proposed scores perform better; C-HDR seems best

C-HDR indeed performs very well, especially for region size when density estimation is accurate and feasible. However, C-PCP and L-CP offer distinct practical advantages in specific scenarios:

• C-PCP: Requires no explicit density estimation or likelihood evaluation, only samples from a generative model. This makes it applicable to models like diffusion models or GANs where densities might be intractable or unavailable, situations where C-HDR cannot be used.
• L-CP: Achieves asymptotic conditional coverage (ACC) with significantly lower computational cost (Figure 6, often >100x faster) compared to C-HDR and C-PCP, as it avoids per-instance sampling or complex density calculations. Both C-PCP and L-CP achieve asymptotic conditional coverage (ACC), unlike DR-CP/PCP/HD-PCP/STDQR/M-CP/CopulaCPTS. Their region sizes are competitive, generally only surpassed by C-HDR (if applicable) and DR-CP (which lacks ACC guarantees). Table 1 and the Conclusion summarize these trade-offs.

Suggestions

Thank you, these are excellent suggestions for improving clarity. We have implemented them as described in this linked PDF and will incorporate other comments in the final version.

Questions:

1. Challenges in high-dim CP: Primarily, the lack of a natural ordering (unlike 1D) makes simple extensions of univariate methods (like CQR directly) difficult. This motivates methods based on joint density (DR-CP, C-HDR), samples (PCP, C-PCP), or latent spaces (L-CP, STDQR) to capture complex dependencies and define multivariate regions.
2. DR-CP empty contours (Fig 2): This occurs when, for a given $x$ and a desired coverage level (e.g., $x=1$ and $1-\alpha$ = 0.2), no region in the output space achieves a density value $\hat{f}(y|x)$ high enough to be less than or equal to the constant threshold $-\hat{q}$. This visually demonstrates DR-CP's lack of conditional coverage. Methods with asymptotic conditional coverage (C-HDR, C-PCP, L-CP) adapt their thresholds or regions based on x and avoid this issue asymptotically.
3. Claim of new classes: We introduce generalizations of existing univariate concepts to the multivariate setting, forming broader classes:

• CDF-based: Generalizes the univariate HPD-split/C-HDR score [31] to work with any base multivariate conformity score $s_W$ (Eq. 11), leading specifically to C-PCP when $s_W=s_\text{PCP}$.
• Latent-based: Generalizes univariate Distributional CP [13] to multivariate outputs using any invertible conditional generative model Q and any latent distance function $d_\mathcal{Z}$ (Eq. 14), leading to L-CP. Feldman et al. [20] also uses a latent space but performs CP on grid samples mapped to the output space, not directly in the latent space like L-CP.

4. Practical takeaway from Prop 1: An interesting practical takeaway follows from the fact that DR-CP and C-HDR are linked in the same way as PCP and C-PCP. Since DR-CP can be shown to asymptotically have the smallest average region size while C-HDR empirically has a smaller median region size, similar observations are expected for PCP and C-PCP. This is verified empirically: PCP has a smaller average region size across all base predictors, while C-PCP has a smaller median region size. We have added this insight to Section 5.3.
5. Distance for C-PCP/L-CP: In our experiments, PCP (and thus C-PCP) uses Euclidean distance, and L-CP uses the Euclidean norm in the latent space. Both could be generalized to other metrics.
6. i.i.d. vs exchangeable data: Thank you for pointing this out, the analysis remains valid. We will relax this assumption.
7. Acronyms: C-PCP = CDF-based Probabilistic Conformal Prediction. L-CP = Latent-based Conformal Prediction.

Thank you for your positive evaluation and constructive feedback. We address your questions below.

Comparison of CP under different conditional density estimation (CDE) variants: We agree that evaluating CP methods across different CDEs is important for robustness. We performed extensive experiments (detailed in Appendix G) using a diverse set of CDE models: two normalizing flows (MQF² [33], conditional Glow [54]), Distributional Random Forests [12] (with KDE), and a GMM hypernetwork [23, 8]. Our key findings regarding the relative performance and properties of the compared conformal methods remained stable across these different base predictors, demonstrating the robustness of our conclusions.

Comparison between our normalizing flow approach and the VAE approach of STDQR

Our primary goal was a fair comparison among conformal methods. To achieve this, we standardized the base predictor architecture where appropriate. For STDQR [20], we replaced the original CVAE with a conditional normalizing flow (specifically, MQF² [33]). This ensures that comparisons between STDQR, L-CP, and other methods are not confounded by differences in the underlying generative model architecture (VAE vs. Flow). This adaptation was also motivated by the exact invertibility and direct density evaluation offered by flows, eliminating the noise associated with VAE sampling and the need for directional quantile regression in the latent space, as discussed in Appendix F.3 and recommended by Feldman et al. [20]. While a direct empirical comparison between the two models is outside the scope of this comparative study, our approach ensures a more direct comparison of the conformalization strategies themselves.

Pairing between CP methods and conditional density estimation methods

This is a valid point. Ideally, each CP method could be paired with its optimal CDE. However, for a unified comparative study, this would introduce confounding factors, making it difficult to isolate the performance differences attributable to the CP methods themselves versus the CDE pairings. Our approach was to select a set of strong, representative CDEs (as listed above) and apply them consistently across all applicable CP methods. This ensures a fair comparison of the conformal methods within our unified framework. Appendix G shows that the relative rankings and conclusions are largely consistent across these CDEs, suggesting that our findings are not overly sensitive to a specific CDE choice within this representative set.