Thank you very much for your positive and constructive comments on our paper. Please find below a detailed response to your questions.

Experimental Designs:

• To improve clarity, we will add the target coverage level $1-\alpha=0.9$ in the captions.
• As index of variability, in Table 1, we report the maximum and the minimum of the empirical coverage observed over the 20 replications. A part of the table (which refers to LM) is reported below. The variability (see Range) is not so different for all the methods.

• Some of the simulation studies report also the results of split conformal prediction with target coverage $1-2\alpha$. This implies that the proposed variants (trained at level $\alpha$) and split conformal prediction with target level $1-2\alpha$ guarantee the same teorethical coverage (and this is equivalent of using $\alpha/2$ to obtain coverage $1-\alpha$). In Table 5, for example, the eu-mod-cross method is better in terms of size than split conformal prediction trained at level $2\alpha$. (Standard) cross-conformal prediction at level $2\alpha$ could be added; however, we expect smaller sets if compared to those of split conformal but with the same empirical coverage (as observed for the $\alpha$ level). We discuss more on this in the first point of Other suggestion.

Weaknesses:

See the first point in Other Suggestions.

References:

Thank you very much for pointing out some relevant references. We will add them to the paper.

Other Suggestions:

• Thank you for you comment. The question of which variant of cross-conformal prediction is to be preferred in practice is subtle:

  • If one really needs to have a rigorous $1-\alpha$ guarantee against the worst case distributions and unstable algorithms (for example, if there are downstream decisions that crucially depend on the provided theoretical guarantee), then it makes sense to run our new methods at level $\alpha/2$.
  • If one is only using conformal prediction as "weak guidance" for downstream decisions, and the user is somehow ok with violations of the target $1-\alpha$, then we recommend running our variants of cross-conformal at level $\alpha$ (where its worst case guarantee is $1-2\alpha$ but it achieves coverage in between $1-2\alpha$ and $1-\alpha$).
  • If the situation is in between, where conformal prediction is neither being used extremely rigorously, nor as loose guidance, but one wants $1-\alpha$ coverage for "typical" datasets and algorithms, but is ok with both overcoverage and undercoverage for odd distributions/algorithms, then perhaps the original (modified) cross-conformal is best.
  • If one does not tolerate over- or under-coverage of any sort, and really wants essentially exact $1-\alpha$ coverage, then only split conformal delivers the goods.

• We agree with this suggestion and we will revise the text accordingly to clarify the distinction.

• Thank you for the suggestion, we will consider this change.

• We will add the assumption of exchangeability within the theorem statements, as suggested. Example: "... . If data are exchangeable,

• Theorems 4.4 and 4.7 require that the p-values $P_1, \dots, P_K$ be exchangeable (and the exchangeability of the p-values is shown in Theorem 4.2). However, the assumption of exchangeability is not necessary for proving Theorem 4.6. It is sufficient that $P_1, \dots, P_K$ are valid p-values (i.e., $P(P_k \leq \alpha) \leq \alpha$), and this assumption is satisfied in the case of rank-based p-values. In fact, the rules $2 \times \text{mean}(\mathbf{P})$ (used by Vovk et al. (2018) to prove the coverage guarantee for cross conformal prediction) and $2/(2-U) \times \text{mean}(\mathbf{P})$ are valid under arbitrary dependence, which is a broader condition than exchangeability. We will clarify this in the text.

We appreciate your feedback. Below is our response to your comments

Experimental Designs

We now additionally analyze a real dataset on electricity consumption, where accurate uncertainty quantification is crucial as the supplier’s revenue depends on customer energy use. The dataset contains 35,411 observations and 20 covariates; 30,000 observations are used for training, while the remaining ones are used for testing. The splitting is repeated 20 times, and the algorithm used is a random forest. In this case, methods such as Full CP and Jackknife+ can be computationally expensive, making sample-splitting-based methods preferable. From the table, it can be seen that Cross CP is more efficient than Split CP, producing more informative sets while ensuring valid coverage. See the first point in Questions below for a further discussion.

The column Uneven split represents Split CP, where the training set comprises a $(K-1)/K$ fraction of the data points (ie, Split CP with the same fraction of training data points used by a single round of Cross CP). The last two columns represent cases where p-values are merged using the Bonferroni rule at level $\alpha$ (ie,$K\min(p)$) or at level $2\alpha$ (ie,$K/2\min(p)$). These results are discussed in the 2nd point of the Questions section

References

• Thank you for pointing to this. We will add the paper by Saunders et al
• We will discuss the two references appropriately in the text

Questions

Conclusions

We believe that the methods has practical utility. For instance, when dealing with a large number of observations (and potentially a high-dimensional feature space), some methods, such as full CP and jackknife+, become impractical. One possible approach is to trade off some statistical efficiency for reduced computational cost by using methods based on sample-splitting. However, split CP can be inefficient because it only uses a subset of the data for model training. Cross CP generally improves the efficiency of split CP but can overcover. Our work improves statistical efficiency of Cross CP (at the same computational efficiency) while maintaining its coverage guarantee.

The question of which variant of cross CP is to be preferred in practice is subtle.

• If one really needs to have a rigorous $1-\alpha$ guarantee against the worst case distributions and unstable algorithms (for example, if there are downstream decisions that crucially depend on the provided theoretical guarantee), then it makes sense to run our new methods at level $\alpha/2$
• If one is only using conformal prediction as "weak guidance" for downstream decisions, and the user is somehow ok with violations of the target $1-\alpha$, then we recommend running our variants of cross-conformal at level $\alpha$ (where its worst case guarantee is $1-2\alpha$ but it achieves coverage in between $1-2\alpha$ and $1-\alpha$)
• If the situation is in between, where conformal prediction is neither being used extremely rigorously, nor as loose guidance, but one wants $1-\alpha$ coverage for "typical" datasets and algorithms, but is ok with both overcoverage and undercoverage for odd distributions/algorithms, then perhaps the original (modified) cross-conformal is best
• If one does not tolerate over- or under-coverage of any sort, and really wants essentially exact $1-\alpha$ coverage, then only split conformal delivers the goods

We acknowledge that the conclusions can be improved and will incorporate the above takeaway messages there

Discussion Lei et al

The results in Lei et al are based on the Bonferroni rule, that is not powerful when p-values (or sets) are highly dependent. Indeed, the Bonferroni correction is tightest when the p-values are nearly independent, while the conformal p-values across folds are highly dependent. On the other hand, the rule "twice the mean" used in cross CP is more powerful if p-values are dependent; see Sec.6.1 in Vovk & Wang (2020). For completeness; the above table also reported the use of the Bonferroni rule as merging function. The last 2 columns guarantee coverage of at least $1-\alpha$ and $1-2\alpha$, respectively. However, the methods have coverage near 1. This aligns with Thm.4 in Lei et al, which states that multisplit+Bonferroni produce wide sets (specifically, sets are wider than single split CP). For comparison, we have added split CP with an uneven split. The set size is smaller than that of Split CP but larger than that of Cross CP and variants. Moreover, its Sd is higher. We will add this discussion

Thank you very much for your positive feedback on our paper. We truly appreciate your comments.

Thank you for noticing the typo, we corrected it.

Thank you for your feedback on our paper. Please find below a detailed response to your concerns.

Experimental Designs Or Analyses:

Thank you for the suggestions. Below, we present an example where the experimental setting is identical to the last experiment in Appendix E, but with $K$ set to 5, 10, and 20. In addition, the number of trials is 100. The second table ($K=10$) is similar to the table reported in the paper with 20 trials (this number is used in many conformal prediction works; see, for example, Barber et al. (2021) or Romano et al. (2019)). There is a slight decrease in the empirical size of sets, and a slight increase in variability, of the e and eu-mod-cross methods as $K$ increases. Clearly, nothing changes for the split conformal methods. We will add the example and add more trials for other experiments.

We additionally analyze the case where the number of observations in each fold differs. Since in cross-conformal prediction the $K$ folds serve the same role (being used for both training and calibration), having folds with significantly different sizes would be unwise. This principle applies not only to cross-conformal prediction but also to other methods that randomly partition the data into $K$ folds, such as Hulc (Kuchibhotla et al., 2024) and MoM (Lugosi & Mendelson, 2019). As explained at the beginning of Section 3, this means that the folds can differ by at most $K-1$ observations (with $K$ typically set to 5 or 10). For completeness, we present the results of a simulation study based on the Boston dataset. The setting is the same as described in Appendix E, except that $n=204$ and one fold at random contains 44 observations instead of 40 (a 10% increase). The number of trials is 100. One could, in principle, have four folds with 41 observations and one with 40, but we consider a more 'extreme' setting. The results are presented in the Table, and the conclusions remain qualitatively similar to those reported in Appendix E.

Weaknesses:

We agree that randomization can introduce variability in the prediction sets; however, employing randomized and asymmetric combination rules is the only way to enhance the efficiency of cross-conformal prediction sets while preserving the same (worst-case) coverage guarantee. Furthermore, as noted in Remark 4.8, randomization plays a role at various stages of the data pipeline in both cross and split conformal prediction. As discussed in the paper, in some applications of large-scale deployment involving thousands of daily predictions, improved statistical efficiency may be preferred.

In light of the responses provided, we hope we have addressed your concerns and that you may consider raising the score.

Kuchibhotla, A. K., Balakrishnan, S., & Wasserman, L. (2024). The HulC: confidence regions from convex hulls. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(3), 586-622.

Lugosi, G., & Mendelson, S. (2019). Mean estimation and regression under heavy-tailed distributions: A survey. Foundations of Computational Mathematics, 19(5), 1145-1190.