Generalized Venn and Venn-Abers Calibration with Applications in Conformal Prediction

Q1: By augmenting the dataset with all possible imputed outcomes for the test point whose outcome we wish to predict we are able to convert a point prediction into set prediction, where the width of this set captures epistemic uncertainty in the calibration process. This augmentation approach is the workhorse of the finite-sample distribution-free calibration guarantees of the original Venn-Abers calibration and conformal prediction procedures of Vovk and similarly is a crucial component of our algorithms.

Q2: The proposed algorithm applies to any smooth, bounded loss function, including standard choices such as squared error loss and quantile loss. It also accommodates more specialized losses, such as those used in missing data settings, as well as losses tailored to estimating conditional functionals like the conditional average treatment effect or the conditional relative risk, as discussed in several of our cited works. We will include additional examples of representative loss functions in the revised manuscript to clarify the range of settings where our method applies.

Thank you for the helpful comments.

• Thank you for these thoughtful suggestions. We will add clarification on how the Sherman–Morrison approach can be used to efficiently implement the algorithm. We appreciate the request for more detail on the case of infinite $\mathcal{Y}$, and will revise the relevant sections to more formally explain how analyzing extreme values of $y$ can suffice to determine the range of the Venn prediction set. We also agree that it would be helpful to situate our generalization of calibration within the broader context of elicitable properties. In the revised introduction, we will add background on this connection and clarify how calibration for general losses relates to elicitable properties.

• We will add further details on the computational and implementation aspects of our approach, including how to efficiently handle infinite $\mathcal{Y}$ and how discretizing the model output can be used to approximate Venn–Abers calibration effectively.

Thank you very much for your detailed comments and suggestions. We will incorporate them in the revised version of the paper.

We note our primary contribution is the generalization of Venn and Venn–Abers calibration to arbitrary loss functions. As a secondary contribution, we show that the same techniques yield a multicalibration algorithm with set-valued predictions. Our goal is primarily theoretical: to present a unified framework that generalizes and connects existing methods, including Venn–Abers calibration and conformal prediction.

1. In the revised paper, we will also report multicalibration error over specific subpopulations. In our experiments, we use the $\ell^2$ norm defined in Equation (4), corresponding to regression multicalibration error. This metric, also used in [1] for quantile multicalibration, ensures multicalibration over finite-dimensional classes of covariate distribution shifts—specific subpopulations being a special case. We define the function class in Equation (4) as linear combinations of binary features, one-hot encoded categorical features, and spline-transformed continuous features (with 5 knots). The reported $\ell^2$ error is the norm of the vector of calibration errors across these transformed features. For datasets with binary features, this directly captures multicalibration error over the associated subpopulations.

2. We recognize that the evaluation of our multicalibration experiments can be improved, and we will address this in the revision. To the best of our knowledge, our Venn multicalibration algorithm is the only method that produces prediction sets capturing uncertainty under multicalibration in regression settings, and thus lacks direct baselines. In principle, our approach can be combined with any multicalibrator that outputs empirically multicalibrated point predictions—such as those in the cited references—to generate uncertainty-aware prediction sets. One goal of our experiments is to show how prediction set widths vary across datasets with different sample sizes and feature dimensions. For instance, the \textsc{Comm} and \textsc{Star} datasets yield wider prediction sets due to high feature dimensionality and small sample sizes, reflecting greater uncertainty in the multicalibration process.

3. Thank you for this suggestion. We agree that a clear discussion of calibration quality and appropriate evaluation metrics is important for interpreting and comparing results, and we will include this in the revised version. In the multicalibration experiments, our primary goal is to assess the quality of set-valued predictions in terms of (i) their size and (ii) the calibration error of the oracle multicalibrated prediction guaranteed to lie within the set. Specifically, we aim for multicalibration uniformly over target populations whose density ratios with the source population lie in a linear class derived from a basis transformation of the features. The reported $\ell^2$ multicalibration error measures how well the predictions satisfy the multicalibration criterion across many overlapping subgroups defined by these density ratios. This approach is motivated by prior work in quantile multicalibration [1] for feature-conditional coverage. We will incorporate this clarification in the final version to better guide practitioners and researchers seeking meaningful evaluation metrics for multicalibrated prediction sets.

4. Since the datasets are based on real-world data, we cannot generate independent replicates to compute standard Monte Carlo error estimates. Instead, we averaged all metrics over 200 random train-test-evaluation splits. In the final version, we will report Monte Carlo error estimates reflecting the variation of these metrics across the random splits.

5. In the revised paper, we will also report subpopulation multicalibration errors. The multicalibration definition in Equation (5) extends beyond fixed subgroups to classes of "covariate shifts", following the terminology of [1] on quantile multicalibration. Consistent with this work, we adopt a model- and subgroup-agnostic approach, aiming to approximate multicalibration across many subgroups without requiring explicit specification. In our experiments, we achieve this by calibrating over subgroups whose density ratios with the source population are additive functions of the covariates, implemented via adjustments over additive spline basis functions. While we agree that subgroup-specific calibration is also valuable, our experiments focused on guarantees that do not depend on manual subgroup definition.

(6) In the revised version, we will add background on multicalibration and clarify how our contributions relate to prior work. While multicalibration is an important aspect, we note our primary focus is on calibration.

[1] Gibbs, Isaac, John J. Cherian, and Emmanuel J. Candès. "Conformal prediction with conditional guarantees." arXiv preprint arXiv:2305.12616 (2023).

Thank you for the thoughtful questions and suggestions. We will take incorporate your suggestions in the revised version of our manuscript. We clarify the efficiency and practicality of our method below.

1. Efficient Approximation and Scalability
   The algorithm can be efficiently approximated using the approach described in Section 3.4 of [1] for Venn-Abers calibration with squared error loss, which we also adopt in our implementation. The key observation is that the Venn-Abers algorithm only depends on the test point features through the original model prediction (which is crucially a one-dimensional quantity). As a result, we can discretize the one-dimensional prediction space—e.g., by binning into a grid of 200 bins—and run the algorithm once per grid point. Isotonic regression can be computed efficiently using regression trees with monotonicity constraints, such as those available in XGBoost. Across all datasets (each with tens of thousands of data points), computing the approximate Venn-calibrated sets for all points takes under a minute. This method scales well to large datasets via XGBoost. The isotonic regression step is computationally lightweight and can be parallelized if needed.

2. Efficiency at Inference Time In our implementation, we precompute the Venn-Abers prediction sets over a discretized grid (200 bins) of model predictions. At inference time, we use nearest neighbor matching or linear interpolation to produce calibrated prediction sets, avoiding any calibration computation at run time. The approximation error from this approach is negligible in practice, likely due to the piecewise constant nature of both the isotonic regression solution and the Venn-Abers sets.

3. Clarification on Disjointness We are unsure why disjointness would be required. The Venn-Abers prediction set is defined as the set of predictions obtained when calibrating with a new data point and all possible imputed outcomes. There is no requirement or expectation for this set to be disjoint.

4. Improving experiments We will add additional baselines to the conformal prediction experiments, including the conditional conformal prediction method of [1] and approaches based on alternative conformity scores, such as the normalized absolute residual error. We will also include error bars for our evaluation metrics to account for variability across the 200 train–validation–test splits over which the results are averaged.

[1] van der Laan, Lars, and Ahmed M. Alaa. "Self-calibrating conformal prediction." arXiv preprint arXiv:2402.07307 (2024). [2] Gibbs, Isaac, John J. Cherian, and Emmanuel J. Candès. "Conformal prediction with conditional guarantees." arXiv preprint arXiv:2305.12616 (2023).