Backward Conformal Prediction

First, we would like to thank the reviewer for their time. We are confident in our approach and hope that our response will clarify our aims, the interpretation of our results, and comparison to classical statistical methodology.

• The contribution of this paper is to provide an estimator for E[α]. However, the main results for backward conformal prediction are provided by Gauthier et al. All the paper does is provide an estimator, which is sadly not so meaningful due to the unknown approximation error caused by the first-order Taylor approximation in lines 134-135.

Our method indeed builds on the use of e-values in conformal prediction, in particular drawing from the work of Gauthier et al. However, that paper did not focus (at all) on obtaining marginal guarantees. In contrast, this is the central contribution of our work. We emphasize that our approach to providing the marginal guarantees (specifically the expected coverage E[α]) is entirely novel. This contribution is crucial in practice, as it enables practitioners to understand the actual coverage they can achieve with this method and make informed decisions accordingly.

Regarding the final part of the comment, the Taylor expansion is never used to approximate any quantity; it is solely a technical tool to establish the coverage property: Pr(Y_test \in C(X_test)) \ge 1 - E[α]. Both the experiments from Gauthier et al. and our own results support and justify the validity of this approximation. We find empirically in a range of experiments that this coverage inequality always holds in practice.

• The experiments show that the estimated value α^LOO is close to the true value E[α]. However, this does not mean that the theoretical bounds here are tight. Note that given the theoretical results, we are going to trust prediction sets with 1-α^LOO-R(n) \ge \tau, which is a distribution-free result. So the experimental results are not sufficient to show the effectiveness of Backward Conformal Prediction as claimed in lines 244-245.

We are somewhat surprised by the comment about tightness, given that the 1/\sqrt{n} and 1/n convergence rates that we obtain for the mean and the variance, respectively, are ubiquitous in statistics. These rates arise from fundamental properties of random variables, probability theory, and the general structure of estimation problems. While one could pursue analyses that establish lower bounds, that effort would duplicate statistical literature and be far beyond the scope of the paper.

We certainly agree that our results are distribution-free, and we view this as a key strength of our method. We are puzzled as to how the reviewer could view this property negatively, especially since the experimental results align closely with the theoretical guarantees we establish.

We would appreciate it if the reviewer could clarify what they intended by this comment. We would be happy to provide further explanation if it would help clarify our approach and make our contributions in the paper more understandable.

• In conclusion, compared with conformal prediction (CP), I do not see the necessity of studying backward conformal prediction (BCP)

The CP and BCP methods are fundamentally different and are designed to address two distinct problems. BCP is not intended to replace CP, but rather to be used in situations where the practitioner prioritizes control over the size of the prediction sets.

• Since the approximation error of the first-order Taylor approximation in lines 134-135 is not controlled, BCP can only control the prediction set size, but not the coverage level.

We do not claim that BCP controls the coverage level. Once again, the Taylor approximation is only used to establish a coverage inequality, and it is consistently satisfied in practice.

Our method, however, does allow for the estimation of coverage. In a sense, BCP even provides more flexibility than CP: while CP controls the coverage but offers no way to estimate the size of the prediction sets, BCP explicitly controls the size of the prediction sets and allows for coverage estimation.

• It is because although BCP can control the size, its rules on the coverage are ambiguous due to the unknown approximation error of the Taylor approximation

Again, the Taylor approximation is only used to derive the coverage inequality; moreover, we observe that this inequality always holds in practice, which is consistent with previous results from Gauthier et al. Furthermore, our estimator allows for the estimation of the coverage guarantees.

• By the way, in situations where errors can cause serious consequences, controlling the accuracy (coverage) is obviously more important than controlling the set size. Furthermore, the spirit of confidence sets prediction (which includes conformal prediction) is to control the coverage level but not the set size, just like the spirit of hypothesis testing is to control the type-1 error but not the type-2 error.

The reviewer is referring to classical statistical practice, dating back to Neyman and Pearson and to Fisher. But just because it is classical does not mean that it is always the preferred approach. Indeed, in the paper, we provided motivating examples, for example in medicine, where controlling the size of prediction sets is crucial. In such examples, the key quantity to control is the time devoted to having experts chase down possible leads that arise from a statistical analysis. Obtaining a large uncertainty set can be unrealistic and indeed useless if requires too much time for experts to perform a follow-up verification to whittle down the possibilities. It may be preferable to allocate expert time/effort in advance and to proceed if the estimated coverage is sufficiently large. That’s what our method makes possible.

Our method is not meant to replace existing classical approaches where the coverage level is fixed in advance. Rather, it is complementary and orthogonal, offering a new perspective on uncertainty quantification.

One may view it as surprising that the statistical literature has focused entirely, for several decades, on controlling coverage, with set size a secondary consideration. It is surprising but true. But perhaps it is because the shift to the inverted logic of our paper is made possible by recent developments in the use of e-values and their associated post-hoc guarantees. This makes it possible to obtain theoretical guarantees even when the coverage level is data-dependent, something that is not possible with p-values.

To say this another way, classical statistical methods have been built around a focus on the control of type I error, typically with a pre-specified, data-independent significance level α and/or p-values. Generations of statisticians and applied researchers (particularly in medicine) have criticized this focus. Accordingly, the controversy around p-values persists. Our work provides one way to escape the tyranny of p-values and to solve inference problems in a new way that is complementary to classical methodology.

We would like to thank the reviewer for giving us the opportunity to clarify the nature of our method relative to classical statistical practice. Our revision will make this comparison more clear and emphasize that our approach is novel, for interesting reasons.

We thank the reviewer for their feedback and continued engagement in the rebuttal process, as well as for their thoughtful comment that helps clarify the decision-making benefits of our method. We have enjoyed the interaction.

First, we are pleased to see that the reviewer recognizes the statistical strengths of our approach, particularly from the perspective of the Pareto frontier. We have also added a comment on this topic in our response to reviewer i8Nu, and we invite the reviewer to refer to it for further details.

The reviewer also raises an important point regarding the accounting aspect of the method, suggesting that CP might be more appropriate in this regard since it does not rely on approximations. We will address the reviewer’s broader concern holistically, rather than dissecting it point by point, in line with the reviewer’s stated preference. We believe that the comparison made by the reviewer is somewhat unfair and inaccurate, even from an accounting perspective, as BCP provides stronger guarantees. Let us explain.

First, we recall that conformal prediction provides marginal guarantees. That is, for any given point, conformal prediction yields a set with population-level reliability, not individual certainty. So for a new patient, neither CP nor BCP can guarantee coverage on that individual case. Both methods offer guarantees of the form: "I know that over the long run, this method will include the true label at least $1-\alpha$ (or $1-\mathbb{E}[\tilde{\alpha}]$ in the case of BCP) fraction of the time, but I don’t know anything specific about this particular case." This limitation of conformal methods has been the subject of growing attention in the literature, which now explores conditional guarantees (see, for example [1]). But in their standard form, conformal methods provide only marginal guarantees.

For this reason, we believe it is not appropriate to evaluate the method based on a single example. It is more appropriate to assess the behavior from a marginal, population-level perspective. Accordingly, we do not frame “rejection” at the level of individual predictions. Instead, in our paper, rejection refers to rejecting an algorithm configuration as a whole. For instance, if a size constraint rule is too restrictive and leads to a predicted coverage ($1-\alpha^{\rm LOO} \approx 1-\mathbb{E}[\tilde{\alpha}]$) that is too low, then the practitioner may reject that algorithm because the marginal guarantees are insufficient for their needs. They may then choose a less restrictive size constraint rule to achieve higher average coverage.

In summary, decisions in conformal prediction are made at the population level, not at the level of single instances. We hope this clarifies our perspective and would be happy to add a clarifying remark in the final version of the paper if the reviewer feels it would improve the reader’s understanding.

Furthermore, we believe that even from an accounting standpoint, BCP provides stronger guarantees. We summarize the benefits of each method below, using the reviewer’s helpful "control"/"calculate" terminology, and distinguishing between individual-level, validity across all samples, and marginal guarantees (which, as discussed above, are central in conformal prediction):

• For CP:

Coverage Guarantee: control (across all samples, corresponds to $\alpha$),

Set Size Guarantee: calculate (at the individual level, corresponds to $|C(x)|$),

Marginal Guarantee: none.

• For BCP:

Coverage Guarantee: calculate (across all samples, corresponds to $\tilde{\alpha}$),

Set Size Guarantee: control (at the individual level, corresponds to $\mathcal{T}(x)$),

Marginal Guarantee: estimation of average coverage (marginal, corresponds to $\mathbb{E}[\tilde{\alpha}]$)

Thus, even from the perspective proposed by the reviewer, which differs from the Pareto viewpoint, BCP still provides stronger guarantees. This is why we believe BCP to be preferable when one cares about both coverage and set size. BCP provides more information than CP, which ultimately allows for better statistical decision-making.

We also want to emphasize that we do not claim BCP is intended to replace CP, but rather to complement it, even if we believe that from both the reviewer’s perspective and the Pareto frontier perspective, BCP offers stronger and more practical guarantees. We believe BCP is particularly useful in settings where controlling set size matters, such as in medical applications.

We hope this response addresses the reviewer’s concerns, and we would be happy to clarify further if any questions remain.

References: [1] Gibbs, Isaac, John J. Cherian, and Emmanuel J. Candès. "Conformal prediction with conditional guarantees." Journal of the Royal Statistical Society Series B: Statistical Methodology (2025): qkaf008.

We would like to thank the reviewer for their detailed review highlighting the strengths of our paper, and we are grateful to know that they appreciated the work. We are also pleased to see that the reviewer found the paper to be well written, clear, original, and addressing an important problem. We will provide a few additional comments on the strengths and questions raised by the reviewer, which are closely aligned with those of reviewer i8Nu.

• The idea in this paper is pretty novel. To the best of my knowledge, I did not find papers similar to this one, which starts from the constraints of the confidence interval

This idea is indeed novel, and we are pleased to see that the reviewer shares our enthusiasm for our new method, which quantifies uncertainty based primarily on a size constraint. It is a promising approach that we believe has potential applications beyond the scope of this paper.

• I am convinced that the constraints of the confidence interval are indeed important in many fields like medicine

We appreciate that the reviewer shares our view on the importance of our method, particularly in fields such as medicine, as illustrated in the paper. We note that reviewer i8Nu agrees on the usefulness of size-constrained uncertainty quantification. However, we are surprised to see that reviewer A3K2 does not share this common enthusiasm.

• The authors further provide theoretical and empirical evidence to validate their findings

We thank the reviewer for highlighting the theoretical contribution of this paper, as well as the experiments that validate our theoretical findings.

• Could the authors provide evidence on why this paper uses conformal e-prediction as the basis? Is it necessary for the backward conformal prediction? What if we directly use other conformal prediction techniques?

We thank the reviewer for their question, which provides an opportunity to highlight the importance of using conformal e-prediction in this context.

First, traditional conformal prediction methods are not designed to control the size of prediction sets, which can be problematic in certain applications (medicine, for instance). We propose a method that is dual to conformal prediction, shifting the focus from controlling coverage at a fixed level to prioritizing control over the size of prediction sets. Our approach is not intended to replace conformal prediction; rather, we believe it complements it and may be more suitable in scenarios where controlling the size of prediction sets is critical.

Regarding the use of e-values, their key advantage lies in the post-hoc guarantees they enable. This allows us to retain theoretical guarantees even when the size-constraint rule depends on the data. Simpler alternatives may exist in more restricted settings, as noted by reviewer i8Nu. However, our method is more general and remains valid even when the size-constraint rule is data-dependent.

We would be happy to add a remark in the final version of the paper to clarify why we rely on e-values in this setting rather than on alternative methods that only apply to the simpler case.

• The experiments part might be a little bit weak. More realistic experiments might enhance the scope of this paper.

We demonstrated the validity of our method on a simple binary classification example and experiments on the popular CIFAR-10 dataset. In future work, we would be happy to apply our method to real-world datasets and even deploy it in practical settings, such as medical applications.

We once again thank the reviewer for their detailed review and their enthusiasm for our work.

First, we would like to thank the reviewer for their time and detailed review. We also appreciate all the positive feedback received regarding the presentation, clarity, and originality of the paper. We will respond to all comments in order and also address the main question raised by the reviewer regarding the benefits of using e-values for this problem.

We note that the points made by the review are similar to that of reviewer kQ7Q, and that our response to their review may contain additional complementary explanations that could help guide the reviewer in their comments.

• The idea of providing set-size constrained CP sets is interesting and useful for a set of real-world applications

We appreciate that the reviewer shares our perspective on the value of size-constrained uncertainty quantification. We believe it is indeed a key method with important real-world applications, notably in medicine, as highlighted in our paper. This view is also supported by reviewer kQ7Q. However, reviewer A3K2 does not find this aspect to be of any relevance, which we found somewhat surprising.

• The LOO method to estimate the coverage level sounds novel to me, and might also be of independent interest beyond CP

We are pleased that the reviewer perceives the scope of our new method even though we preferred to highlight its simplicity in our presentation. Indeed, we specifically designed a proof tailored to our problem, based on an analytical study of the quantile function combined with probabilistic inequalities. This proof could indeed be reused as is, adapted, or serve as inspiration for tackling other problems. The LOO method is indeed novel, and we hope it may inspire similar types of proofs in the literature.

• Despite the pleasant characteristics of e-values, I am still not fully convinced why e-values are actually needed in this scenario, at least from the algorithmic perspective

First, we understand the simple algorithm and method proposed by the reviewer. It is indeed possible to obtain theoretical guarantees under the assumption that the data are i.i.d. However, this only works in the simple case where the size-constraint rule does not depend on the data. While we did treat this case in the paper, it was primarily to build intuition for the proof in the more general setting where the size-constraint rule does depend on the data (including all the calibration data), and the confidence level is random. To achieve sound theoretical guarantees in this more general setting it’s essential to make use of the post-hoc validity that is provided by e-values.

To help future readers to understand this (critical, but somewhat subtle) issue, we will make use of the reviewer’s suggested algorithm as a didactic device to compare and contrast to our general approach. In particular, we will add a remark comparing the different approaches in the case where the size-constraint rule is constant.

We would like to thank the reviewer for their useful and insightful comments.

I have read the second response to reviewer A3K2 as well.

"Exactly, we care about both the size and the coverage. Classical statistics cares about both as well, but focuses ..."

This is an interesting perspective. However, it is still not exactly obvious how it is going to be useful. Perhaps it would be useful to think of a Pareto optimal curve where on one axis we have coverage and on the other we (minimum) average set size (possible). The more coverage you demand, it would NECESSARILY lead to a larger average set size. Hence, in principle, this reverse approach is just giving a different parametrization for the same Pareto curve. Which is interesting by itself to me, but the authors response to the reviewer A3K2 looks like a bit of overclaim in terms of the impact of this point of view.

To see why this Pareto curves matter, one has to pay attention that, if the decision maker (e.g. the doctor) is not satisfied with the size of a normal CP set, and wants a smaller one, they would have to necessarily sacrifice some coverage. There is no magic (unless the authors are claiming it, but I dont see it). Now here it comes the question of whether it makes sense from the point of view of a decision maker to sacrifice coverage for a controlled set size. And this DOES need a thorough discussion from a decision making point of view. I am not saying it is not useful, I am just saying it is not entirely obvious, as it is stated by the authors.

I thank the authors for their detailed response. In particular, the point about data-dependent set size control makes sense to me, hence I keep my positive evaluation.

That being said, I have read other reviewers comments, and I believe some of the points raised by the reviewer A3K2 are valid and has to be addressed properly. For instance, the point about approximation error is valid to me. Also, about the importance of set size control CP, although I find it interesting, the use of it in practice is not entirely straightforward. The thing about coverage control is that it has some clear implications for the down-stream decision makers, for instance look at the recent work of [1]. It is not obvious how a set size controlled prediction set should be used or affect the decisions. This needs more discussions.

[1]: Decision Theoretic Foundations for Conformal Prediction: Optimal Uncertainty Quantification for Risk-Averse Agents, Kiyani et. al.