We thank the reviewer for their time and insightful comments. Please find our answers to the points raised below.

The paper is very dense and hard to read. Some toy examples / walk-throughs of the procedure could be quite useful for understanding.The template-building procedure is not described well.

To improve the clarity of the paper, we will add a pedagogical figure that explains the procedure in two steps in the updated manuscript. The first panel illustrates the concept of JER control with an example of template and of calibrated threshold family. The second panel illustrates the intervals obtained with CoJER using the adjusted risk level $\widehat{\alpha} = t_{\lfloor \alpha m\rfloor}$ with $t$ the calibrated threshold family. We will also add pedagogical interpretations for the notions of threshold families and templates.

The matrix dimensions are mentioned to be n x p, but p is never mentioned before. This is quite confusing. (It might be a typo making, and it should be n x m)

Thanks for pointing this out, this is indeed a typo -- the matrix dimensions are $n \times m$. We have corrected this in the updated version of the manuscript.

The authors don't theoretically or experimentally explore the relative tightness of the finite sample FCP bounds depending on aggregation mechanism used.

To address the reviewer's concern, we performed an additional experiment on the 17 openML datasets used in the paper. In this experiment, we compare four possible aggregation schemes: harmonic mean, arithmetic mean, geometric mean and quantile aggregation [1].

We use the setup described in the main text - i.e. $\alpha = 0.1, \delta = 0.1$. Importantly, we first check that the FCP is controlled for all types of aggregation by reporting the FCP event non-coverage as described in the main text. We also compute the (relative) interval width for each aggregation scheme, averaged across 20 splits.

Coherently with the theoretical guarantees obtained in Proposition 4, the FCP is controlled for all four aggregation schemes. In terms of interval tightness, harmonic mean aggregation outperforms arithmetic mean, geometric mean and quantile aggregation consistently.

References

[1] Meinshausen, N., Meier, L., & Bühlmann, P. (2009). P-values for high-dimensional regression. Journal of the American Statistical Association, 104(488), 1671-1681.

We thank the reviewer for their time and insightful comments. Please find our answers to the points raised below.

In Lemma 1, how are ( p_1, \dots, p_m ) defined? Is ( p_j ) equal to ( P(Y_j) ), where ( P(\cdot) ) represents the p-value function?

For each $j \in [[m]]$, we define $p_j := \frac{1}{n+1} \left(1 + \sum_{i=1}^n \mathbf{1}[S_j \le S_i]\right)$ as per Definition 1. We have added this to Lemma 1 in the updated version of the manuscript.

The concepts of threshold families and templates may be abstract for those unfamiliar with this literature. Providing some context on why these notions are introduced and their practical applications could be helpful.

We will add a pedagogical figure that explains the procedure in two steps in the updated manuscript -- we detailed this in our answer to reviewer DTau. In a nutshell, the threshold family can be viewed as a (vector) parameter, and templates are sets of candidate parameters, from which a candidate ensuring JER control (and hence FCP coverage) is chosen by the calibration algorithm.

Regarding the "parametric" nature of the Gazin et al. bound, we agree that the use of "parametric" can be confusing here. Indeed, the paper of Gazin et al. (2024) considers distribution-free conformal prediction settings. We wanted to emphasize that the confidence envelope of $\widehat{F}_m$ they propose has a fixed shape. Theorem 2.3 of Gazin et al. (2024) is a Dvoretzky–Kiefer–Wolfowitz (DKW) type inequality, proved to hold for the specific (known) dependence between conformal $p$-values. As noted in that paper (see Remark 2.6 therein), this approach can be conservative, which can be addressed by the JER calibration framework. CoJER is fully non-parametric in the sense that the shape of the tempate is itself derived from the joint distribution of $p$-values, and not chosen a priori. Our experimental results show that CoJER is much less conservative than the approach of Gazin et al. (2024) while preserving FCP control.

Explicit construction of the confidence intervals: we thank the reviewer for pointing this out. Once Algorithm 3 is run, we obtain a JER controlling family $t$. Corollary 1 states that, with probability at least $1-\delta$, $\forall \alpha \in[0,1], \quad \operatorname{FCP}(\mathcal{C}(\alpha)) \leq \frac{j(\alpha, \delta)}{m}$ with $j(\alpha, \delta) = \min \{j \in [[m]] : \alpha \leq t_j\}$. In turn, to obtain FCP control at level $\alpha$ with probability $\geq 1 - \delta$ we choose $\widehat{\alpha} = t_{\lfloor \alpha m\rfloor}$. Therefore, the FCP-controlling intervals can be explicitly written $\mathbf{\mathcal{C}(\widehat{\alpha})} = (C_{i, \widehat{\alpha}})_{i \in [[m]]}$ with $C_{i, \widehat{\alpha}} = \left[\hat{\mu}\left(X_{n+i}\right) \pm S_{(\lceil(n+1)(1-\widehat{\alpha})\rceil)}\right]$. The confidence intervals only depend on the threshold family via $\widehat{\alpha}$. We have added this explanation to the updated version of the manuscript.

Template optimality: while our experiments unambiguously show that CoJER leads to tighter FCP control than the bounds of Gazin et al, we currently do not have theoretical support for this result. In fact, even the formal definition of an optimal shape is not trivial, and we have left this exciting perspective for future work.

Does the model aggregation scheme preserve the coverage properties of CP? Does this follow from JER control?

CoJER does not ensure a marginal coverage guarantee for each test observation when $m>1$. Indeed, we argue that this type of guarantee is not interpretable in the transductive case considered here. Instead, CoJER offers a strong probabilistic guarantee (FCP control) over the entire set of $m$ test observations. However, note that for a single observation ($m=1$), FCP control is equivalent to the marginal coverage guarantee offered by SCP.

Significance level and comparison to the Benjamini-Hochberg procedure: indeed, the reviewer's intuition is correct. CoJER builds a JER controlling family $t$ and outputs an adjusted level $\widehat{\alpha} = t_{\lfloor \alpha m\rfloor}$ for which the confidence intervals $C(\widehat{\alpha})$ controls the FCP at level $\alpha$. This is markedly different from the Benjamini-Hochberg (BH) procedure. First, BH outputs a rejection set for which there is a statistical guarantee. In the transductive setting of conformal prediction, we are interested in obtaining a guarantee for all test points simultaneously and not only for a certain subset. This renders BH and other similar methods unapplicable in this context. Second, BH offers a guarantee on the expected proportion of False Discoveries and not in probability.

Title: we agree that the title is too vague and doesn't clearly distinguish the contribution. We propose to rename the paper False Coverage Proportion control for Conformal Prediction. To this end, we have sent a message to the PCs of the conference.

We thank the reviewer for their time and insightful comments. Please find our answers to the points raised below.

However, one potential limitation is that the paper primarily evaluates performance on tabular regression datasets. The applicability of CoJER to classification problems or high-dimensional deep learning models is not explored in depth.

While our experiments focus on tabular regression datasets, we would like to emphasize that CoJER is fundamentally agnostic to the specific predictive setting as long as the transductive assumption holds (i.e., access to many test points). This stems from the fact that CoJER operates purely on conformal p-values inherited from the CP framework. As such, the method is applicable to classification tasks and other model families, including high-dimensional deep learning models, provided conformal $p$-values are available.

No ablation studies to test different parameter choices.

The main parameter of CoJER is the aggregation function. We have performed an additional experiment to compare four possible choices: harmonic mean, arithmetic mean, geometric mean and quantile aggregation. In this setting, harmonic mean aggregation outperforms arithmetic mean, geometric mean and quantile aggregation consistently. Please see our answer to reviewer DTau for all experimental details.

Monte Carlo estimation reliance—more discussion on computational trade-offs is needed. How does CoJER perform on classification problems?

In our setting, we see the reliance on Monte Carlo estimation as a strength rather than a weakness, since it allows JER control with arbitrary precision at a small computation cost. Sampling from $P_{n,m}$ is done in $O(n + m)$. Moreover, this is done once and for all for given values of $n$ and $m$.

What is the computational overhead compared to standard SCP?

In the transductive setting considered in this paper with $m$ test points, both SCP and CoJER require $O(m n \log(n))$ to obtain $m$ conformal $p$-values (each of them requires sorting $n$ conformity scores). For FCP control with $B$ MC samples, CoJER additionally requires $O(B m \log(m))$ for template generation and sorting, and $O(B m (\log(m) + \log(B)))$ for calibration using binary search.

Therefore, neglecting the logarithmic terms for simplicity, the complexity of standard SCP is $O(m n)$ and that of CoJER is $O(m (n+B))$, where $B$ is the number of MC samples (which is user-defined does not depend on $n$ or $m$). In particular, the complexities are of the same order if $B$ is chosen to be of the same order as $n$.

The paper provides a comprehensive review of prior work, but could benefit from including: Adaptive Conformal Inference: Angelopoulos et al. (2023) propose an adaptive method for controlling conformal prediction widths, which may be relevant for understanding how CoJER adapts to different datasets. Resampling-based Conformal Methods: Recent work on Jackknife+ (Barber et al., 2021) explores alternatives to split conformal prediction, which could be useful for comparison.

Both of these approaches provide marginal risk control, i.e. for a single test point. As argued above in our reply to reviewer 8Xty, such approaches could be leveraged to control the False Coverage Rate (FCR) but they do not provide FCP control in probability. Therefore, they are not more relevant than SCP as competitors for our method.

We thank the reviewer for their time and insightful comments. Please find our answers to the points raised below.

However, the evaluation setup raises concerns. Since SCP does not ensure FCP control, it should not be included in interval length comparisons [...] including additional baseline methods or alternative strategies for FCP control, would strengthen the evidence supporting this statement.

To the best of our knowledge, the only existing approach that explicitly targets FCP control in CP is the work of Gazin et al. We would be glad to include additional baselines if others are brought to our attention, and we welcome any suggestions in that regard.

We agree that a fair comparison of interval lengths should ideally be restricted to methods that control the false coverage proportion (FCP). We chose to include SCP, despite it not controlling the FCP, to illustrate that CoJER provides substantially stronger coverage guarantees with only a modest increase in interval length.

The paper lacks a dedicated related work section [...] other risk-control methodologies should have been explored—particularly conformal risk control [1] and risk-controlling prediction sets [2]."

For a new test point $(X_{n+1},Y_{n+1})$, SCP provides a confidence interval $C_{\alpha}\left(X_{n+1}\right)$ with the following guarantee: $\mathbb{P}\left[Y_{n+1} \notin C_{\alpha}\left(X_{n+1}\right)\right] \leq \alpha$. The two approaches mentionned by the referee extend SCP as follows:

• Conformal risk control replaces marginal miscoverage by a general loss $\ell$ , aiming for the guarantee: $\mathbb{E} \left[\ell(C(X_{n+1}, Y_{n+1})\right] \leq \alpha$
• The approach in [2] provide the same guarantees as SCP, in the case of set-valued prediction.

As such, these approaches still provide marginal risk control for a single test point. Our paper focuses on the transductive setting, where $m$ such test points are available. In this setting, while the marginal guarantees could be leveraged to control the False Coverage Rate (FCR), they do not provide FCP contol in probability. Therefore, they are not more relevant than SCP as competitors for our method. We have added text to the section 2.1 "Split conformal prediction" of the manuscript to clarify this important point.

Line 62 (Page 2): How do you define "prediction intervals with a size close to the optimal length"? What constitutes the optimal length in this context?

In this context, we meant that CoJER produces prediction intervals with lengths close to those of SCP, while additionally providing formal FCP control—something SCP does not guarantee. We agree that the term "optimal" could be misleading and have reformulated this point in the updated version of the paper to avoid ambiguity.

What is the impact of $\delta$ on performance? Could setting to a very low value be beneficial?

FCP is controlled with probability greater than $1 - \delta$. As $\delta$ becomes smaller, FCP control becomes increasingly stringent. In terms of performance, this means that decreasing $\delta$ increases interval width. Setting $\delta$ to a very low value (e.g. $\delta = 0.001$) would ultimately lead to a non-informative statement, with FCP control holding with overwhelming probability, for very wide intervals.

The title is too broad; "Tight and reliable conformal prediction" is a common goal in this field and does not clearly distinguish the contribution.

We agree that the title is too vague and doesn't clearly distinguish the contribution. We propose to rename the paper False Coverage Proportion control for Conformal Prediction. To this end, we have sent a message to the PCs of the conference.

Lacks sufficient intuitive explanations for key steps, with some missing motivations (e.g., the choice of the harmonic mean for the aggregation scheme).

Regarding the choice of the harmonic mean, we have performed an additional experiment to compare four possible choices: harmonic mean, arithmetic mean, geometric mean and quantile aggregation. Please see our answer to reviewer DTau for all experimental details.

Limited comparison, as it considers only two methods, one of which does not ensure FCP control, restricting the strength of the empirical evaluation.

Please see our answer above: to the best of our knowledge, the work of Gazin et al. is the only existing approach that explicitly targets FCP control in CP. We would be glad to include additional baselines if others are brought to our attention, and we welcome any suggestions in that regard.

Could you provide relevant examples of multiple test point settings?

A common example of multiple test point setting in CP are real-time systems with batches: In applications like fraud detection or content recommendation, models may process incoming data in batches for efficiency. Another common setup is the offline evaluation of predictive models: before deploying a model, it is often evaluated on a fixed test set.