Conformal Prediction as Bayesian Quadrature

We thank the reviewer for writing a thoughtful and detailed review. We are happy that the review has recognized that the choice of prior can be a headache in practical problems and how our method circumvents this. We also thank the reviewer for reading our proofs. We are glad that the reviewer found them to be enlightening.

Conformal Risk Control vs. Split Conformal Prediction. We thank the reviewer for raising the point about the experiments being run on Conformal Risk Control rather than split conformal prediction. The main reason that the experiments focus on Conformal Risk Control is pragmatic in nature. CRC recovers split conformal prediction when the miscoverage loss is used (i.e. loss = 1 if prediction set/interval does not cover ground truth and loss = 0 otherwise), but allows more general loss functions to be considered. Therefore, in order to investigate more complex loss functions, the experiments focus on CRC. However, please see our next point below which includes results on Split Conformal Prediction.

Additional Experiments. During the rebuttal period we ran additional experiments in a more traditional prediction interval setting with heteroskedastic data. In each of the 10,000 random trials we use 200 calibration samples. We let $X \sim U[0, 4]$ and $Y|X \sim \mathcal{N}(0, X^2)$. Prediction intervals are formed as $[-\hat{\lambda}, \hat{\lambda}]$ where $\hat{\lambda}$ is to be selected by each method. The loss is the miscoverage loss and the target loss is set to 0.1 (i.e. 90% coverage). The maximum allowable risk failure rate is set to 5% (i.e. $\beta = 0.95$). Please note that since the miscoverage loss is used here, Conformal Risk Control and Split Conformal Prediction coincide.

The results indicate that even in the heteroskedastic setting, our method allow more precise control of $\hat{\lambda}$. Unlike RCPS, which is overly conservative, ours produces the shortest prediction intervals while not violating the maximum allowable risk failure rate.

On the nature of conditional coverage. We thank the reviewer for raising this point. Previous work on conditional guarantees have focused on input-conditional guarantees, where the guarantee is conditioned on $X_{n+1} = x$ for all $x$ in the input domain. Guarantees of this nature have been shown to be generally impossible without stronger distribution assumptions. Our guarantees are perhaps better characterized by the term "data-conditional guarantee", where we condition on the set of observed loss values $\ell_{1:n}$. Our experiments demonstrate the practical benefits of this by achieving decisions that produce smaller prediction sets and intervals while not violating the constraint on maximum allowable failure rate. Our guarantees do not rely on strong distribution assumptions that would be necessary to produce an input-conditional guarantee. The distinction between input-conditional and our data-conditional guarantees is an important one and we will be sure to clarify this in the paper by adding a paragraph discussing this point.

Notation. We thank the reviewer for suggesting improvements to the notation. We would be happy to update the notation for clarity. However, unfortunately, the links in the review seem to be pointing to the same paper. If the reviewer would be so kind as to re-post the links in a comment, we would be glad to update our paper accordingly.

We thank the reviewer for writing a detailed review that recognizes the bridging nature of our work and its benefits for various decision-making domains.

Generalizing Conformal Prediction. One of our main contributions is to show how both split conformal prediction and Conformal Risk Control can be recovered by taking the posterior mean of our upper bounding loss random variable $L^+$. Thus, our method illuminates a broader viewpoint and uses this insight to choose $\lambda$ more effectively. We believe this is generally useful perspective that may have broader implications on conformal prediction in general. Nevertheless, we will update the text of the manuscript to clarify the nature of the generalization elaborated here (i.e. oriented towards split conformal prediction and Conformal Risk Control).

Role of the prior. We thank the reviewer for raising this nuanced but important point. Our goal is to show that the Bayesian viewpoint unlocks a richer interpretation compared to previous works, which focus on marginal guarantees that as we have shown in the paper correspond to the posterior mean. For better or for worse, there is still a big gap in the literature between traditional approaches to distribution-free uncertainty quantification, which are predominantly frequentist in nature, and methods like Bayesian quadrature which are firmly Bayesian in nature. Therefore, to draw an explicit correspondence between the two, the dependence on the prior is removed in Section 4.3. The intuition is that that any rational decision maker operating according to the rules of probability, regardless of prior (sufficiently expressive), would agree with the upper-bounding distribution of $L^+$ we derive. Naturally, commitment to a specific choice of prior would lead to tighter distributions over the posterior risk, and in future work we seek to bridge these fields even further by exploring specific choices of priors over quantile functions. We will add a paragraph explicitly discussing these points to the manuscript.

Figure 1. Thank you for raising this point. The interpretation stated in the review is correct: the smoothness does indeed result from the combination of a step function with the distribution over quantile spacings. Our original intention was to illustrate the general idea of Bayesian quadrature in our setting with Figure 1, and then move into the specifics of our method in Figure 2. However, we will update Figure 1 to more clearly signpost the use of the step function, both in the caption and by updating the figure itself.

We thank the reviewer for taking the time to write a thoughtful and detailed review. We are pleased that the review recognized the novelty of creatively combining Bayesian quadrature and distribution-free analysis of random quantile spacings.

We will update the paper to clarify that our guarantees are conditional on the observed calibration data. In essence, our guarantees are probabilistic statments about the risk conditioned on the observed losses (see e.g. the conditioning on $\ell_{1:n}$ in Theorem 4.3 and Corollary 4.4). This stands in contrast to previous approaches which rely on marginalizing over many possible realizations of the calibration losses that were not observed. We will add a paragraph to the paper explicitly clarifying this point.

We thank the reviewer for taking the time to write a thoughtful and detailed review of our work. We appreciate that the review has recognized the "very interesting and novel insights about UQ & decision making" provided by our paper.

We appreciate the desire for additional experiments to provide additional insight about our method. To this end, we have implemented experiments on heteroskedastic data. The findings are largely in line with the results from Section 5 of our paper: our Bayesian interpretation produces prediction intervals that are shorter than baselines while not exceeding the maximum acceptable failure rate.

In each of the 10,000 random trials we use 200 calibration samples. To achieve heteroskedasticity, we let $X \sim U[0, 4]$ and $Y|X \sim \mathcal{N}(0, X^2)$. Prediction intervals are formed as $[-\hat{\lambda}, \hat{\lambda}]$ where $\hat{\lambda}$ is to be selected by each method. The loss is the miscoverage loss and the target loss is set to 0.1 (i.e. 90% coverage). The maximum allowable risk failure rate is set to 5% (i.e. $\beta = 0.95$). Please note that since the miscoverage loss is used here, Conformal Risk Control and Split Conformal Prediction coincide.

The results indicate that even in the heteroskedastic setting, our method allow more precise control of $\hat{\lambda}$. Unlike RCPS, which is overly conservative, ours produces the shortest prediction intervals while not violating the maximum allowable risk failure rate.