CBMA: Improving Conformal Prediction through Bayesian Model Averaging

Thank you for your constructive feedback and for identifying areas where our manuscript can be improved. Below, we provide point-by-point responses to address the concerns you raised. However, we would like to clarify that many of the points you raised are not inherent weaknesses of our method but rather questions or requests for clarification, which we have addressed. We believe it would be unfair to frame these questions as weaknesses to diminish the value of our contribution.

Weakness 1: We respectfully disagree with your criticism that the novelty of our work is limited to simply combining existing methods. While our framework does build upon the well-established Bayesian Model Averaging to enhance conformal prediction, this integration represents a significant and novel contribution, as recognized and highlighted by Reviewer 3y9y. Specifically, our work is the first to provide theoretical guarantees that ensure the resulting conformal prediction sets are optimally efficient, achieving the shortest expected prediction intervals under model misspecification.

Weakness 2: Yes, the Bayesian conformity score is needed in our CBMA framework. As we discussed in the paper (lines 184-186), the Bayesian posterior predictive density is the optimal choice for the conformity score. Accordingly, we constructed our framework around this score. This choice enables us to derive the efficiency guarantee in our Theorem $2$, which ensures that the prediction sets produced by our framework are optimal in terms of their expected size (minimized prediction interval length) while maintaining the required coverage probability.

Weakness 3: As outlined in Equation (7) of the paper, averaging the Bayesian conformity scores associated with different models is mathematically equivalent to computing the Bayesian average of the conformity scores from individual models. Since these two approaches yield the same result within our framework, we are unsure about the distinction implied in your question. We would appreciate further clarification to address your concern effectively.

Weakness 4. While our CBMA framework relies on the standard assumption of infinite sample sizes to establish theoretical guarantees for efficiency and coverage, we emphasize that this is a well-established practice in Bayesian model averaging literature for deriving rigorous theoretical properties. This approach is necessary because Bayesian inference does not inherently align with the frequentist notion of repeated sampling, where guarantees are evaluated under repetitions of the experiment. Instead, Bayesian methods focus on updating beliefs about parameters given observed data. The infinite sample size assumption provides a bridge for deriving frequentist-style guarantees, such as efficiency and coverage, in an asymptotic framework.

However, we demonstrate empirically that CBMA performs effectively even with finite sample sizes. In particular to address your concern, we added Figure $1$ showing the convergence of the mean CBMA prediction set sizes to the optimal prediction set sizes (corresponding to the true model). Convergence is observed with $n=200$, indicating that CBMA achieves efficiency with a finite sample size. Additionally, we included a real data example (California housing data) in the revised Section 5.3. This example shows that CBMA produces shorter prediction intervals than other baseline methods at $n=150$. Notably, Bayesian credible intervals tend to over-cover the data and result in wider, less efficient prediction sets at similar sample sizes.

Case of approximately correct models: As demonstrated in another experiment, even when the true model is not included in the model space, CBMA demonstrates robustness against such model specification. In such cases, theoretical analysis suggests that CBMA converges to the model closest to the true model in terms of Kullback-Leibler (KL) divergence. This ensures that CBMA maintains its coverage guarantee and produces near-optimal prediction sets if a model exists within the KL divergence neighborhood of the true model. We have added detailed discuss of this observation in the revised version.

In summary, while our theoretical results assume infinite sample sizes, the empirical results confirm that CBMA remains highly effective in finite-sample scenarios, offering both robust coverage and efficiency advantages over Bayesian intervals.

Weakness 5: Thank you for the suggestion. There is no Theorem 3 in the paper, but we believe you are referring to Theorem 2. In the revised paper, we have updated theoretical result (Theorem 2) that integrates original Theorem 2 and Remark 1, presenting a unified result for the optimality of prediction sets. As you suggested, this enhancement aligns with the goals of the paper and provides a more comprehensive theoretical foundation.

Thank you for your constructive feedback and for highlighting areas where our manuscript can be improved. Below, we provide point-by-point responses to address the weaknesses you noted.

Weakness 1: We respectfully disagree with your criticism that the novelty of our work is limited to simply combining existing methods. While our framework does build upon the well-established Bayesian Model Averaging to enhance conformal prediction, this integration represents a significant and novel contribution, as recognized and highlighted by Reviewer 3y9y. Specifically, our work is the first to provide theoretical guarantees that ensure the resulting conformal prediction sets are optimally efficient, achieving the shortest expected prediction intervals under model misspecification.

Weakness 2: Thank you for the suggestion. We have renamed subsection 5.2 to ``Approximation using Hermite Polynomials" for better clarity.

Weakness 3: The number of models used follows the setting $K = \lfloor 3 n_{\text{train}}^{1/3} \rfloor$, as mentioned in Line 425. For $n = 50, n_{\text{train}} = 30$, resulting in $K = 9$. These settings align with those in Lu & Su (2015). But, to avoid the confusion and for the sake of completeness, we have added results for model 10 and model 11 for sample size $n = 50$ in the revised version.

Weakness 4: We appreciate the feedback and have corrected the typographical errors, including the missing equation reference and typos before Section 2.

Question 1: We have standardized the term ``prediction set" throughout the paper. While the prediction sets in our experiments are intervals, the theoretical results apply to prediction sets in general.

Question 2: We will include execution time details in our simulation experiments to construct various prediction sets (conformal Bayes, Bayes prediction, BMA prediction and our CBMA ) in the Appendix where we provide additional details of our experiments. For the real data example, we will discuss such execution times in the main text to provide times costs between these different approaches.

Question 3: Following your suggestions, we have added the new experiments in Section 5.1 in the revised paper, with consistent experimental settings with other experiments ($\alpha = 0.2$, train-test split of $60-40\%$), which will improve the clarity and allow comparisons across experiments.

References:

Xun Lu and Liangjun Su. Jackknife model averaging for quantile regressions. Journal of Econometrics, 188(1):40–58, 2015.

Thank you for your constructive feedback. Below, we provide point-by-point responses to address the weaknesses you noted.

Weakness 1: You are correct that conformal prediction based on model averaging has gained attention, as the papers that you provided demonstrate how incorporating model averaging can help reduce the volume of prediction sets through experiments. However, none of these prior works provide a theoretical guarantee. Our work is the first to establish such a guarantee, specifically proving that the resulting conformal prediction intervals converge to the optimal level of efficiency. As Reviewer 3y9y noted, our approach is the first to combine ``multiple models'' in conformal framework through Bayesian model averaging, marking a significant and novel contribution. For this reason, we believe our contribution has been accurately stated. Thank you for providing the references. We have acknowledged these prior works on conformal model averaging in Section 2.5 and highlighted their limitations in the revised paper.

Weakness 2: We appreciate your feedback. Our simulations were intentionally designed to validate the theoretical guarantees of our method and provide comparisons with other approaches in a simple yet illustrative manner, which allows readers to easily recognize that the candidate models are mis-specified, yet our proposed method consistently outperforms others by producing the shortest prediction intervals while maintaining coverage probabilities close to the nominal level. To further demonstrate the practical utility of our method, we have added a real data example in the revised version of the paper and showed that CBMA works effectively. Finally, to improve the presentation of results, we have added plots of comparisons for mean lengths and coverages for prediction intervals in our simulation studies.

Question 1: Following your suggestion, we have thoroughly reviewed the relevant literature, including your recommended papers, Linusson et al. (2020) and Yang and Kuchibhotla (2024), as well as several others in Section 2.5. We have expanded our discussion to provide a more comprehensive discussion of existing methods, highlighting their limitations and how they compare to and differ from our approach. We believe this expanded discussion addresses your concerns and clarifies the unique contributions of our work in relation to the existing literature.

Question 2: We appreciate your thoughtful feedback and understand your interest in exploring more challenging ``misspecification'' cases, such as the AR2 model. However, conducting such experiments may not be necessary in this context, as it is well-known in the literature on Bayesian Model Averaging (BMA) for time series data that if all candidate models fail to account for temporal dependence, BMA will lead to biased estimates and underestimated uncertainty. In other words, efficiency gains are only achievable when the misspecified models still capture temporal dependence to some extent. However, models with temporal dependence structure violate the exchangeability condition required by our method. To address your concern that our method may have limited applicative interest, we have added new experiment to evaluate the performance of our approach on real dataset, where the true model is unknown. This represents a fair assessment of our method's effectiveness, as it mirrors real-world scenarios where all methods may potentially mis-specify the true model. These new results are included in the revised paper.

Question 3: We appreciate your suggestion, as it offers an insightful way to evaluate whether the prediction intervals produced by our method are likely the shortest among all methods, when not knowing the population truths. Specifically, Scheffé's method tests whether the expected lengths of prediction intervals (i.e., the population truths) differ across methods. However, we have already demonstrated in Remark 1 of the paper that the expected interval lengths are theoretically guaranteed to be the shortest under our method. Since the population truth is already known in this context, there is no need to rely on hypothesis testing, which is inherently subject to type I and type II errors and could lead to incorrect conclusions about the population. In conclusion, our theoretical guarantee makes additional hypothesis testing unnecessary.

Revisit Question 1

Thank you for your feedback. As noted in the manuscript and our responses, we conducted a thorough review of relevant works and highlighted either their distinct goals or their lack of theoretical guarantees compared to ours. We also appreciate Reviewer 3y9y pointing out another recent unpublished paper by Liang et al. (2024; https://arxiv.org/abs/2408.07066). However, as we clarified to Reviewer 3y9y, Liang et al. (2024) addresses a completely different goal. Their work focuses on constructing a finite-sample valid conformal prediction set while selecting a model that minimizes the size of the prediction set from a given family of pre-trained models. In other words, their efficiency guarantee relies on knowing the true model.

In contrast, our work provides optimally efficient conformal prediction sets without requiring knowledge of the true model. Our efficiency guarantee holds even under potential model misspecification. This critical distinction was acknowledged by Reviewer 3y9y, who raised their score from 5 to 6, recognizing the unique strengths and contributions of our CBMA framework.

Revisit Question 3

As noted in our responses, we appreciate your suggestion on using hypothesis testing to determine whether the prediction sets obtained using our CBMA framework is the shortest with finite sample size. However, such an approach does suffer from type I and type II error, like all hypothesis tests do. We do acknowledge that our theoretical guarantees on minimal size are asymptotic; in our original submission, we already empirically assessed the finite-sample performance of our CBMA framework in the original submission. These results demonstrate that CBMA performs effectively in finite samples, producing shorter prediction intervals than baseline methods while maintaining valid coverage.

That said, we have taken your suggestion seriously and expanded our empirical evaluation in the revised paper to include both additional simulations and a newly added real-world example. Instead of focusing on hypothesis testing, we have added figures illustrating the convergence of the expected sizes of CBMA prediction sets to the optimal prediction set sizes (corresponding to the true model). In summary, convergence is observed at $n=200$ in simulations and $n=150$ in the real-world example, demonstrating that CBMA achieves optimal efficiency with finite sample sizes.

Lastly, while your request regarding the rate of convergence is beyond the primary scope of our work, we acknowledge its importance. To address this concern, we note that Equation (7) suggests that the rate of convergence corresponds to the convergence rate of posterior model probabilities, a well-studied topic in Bayesian literature (see, for instance, Rossell, 2022).

We hope this additional clarification and empirical evidence address your concerns.

Reference:

Rossell, D. (2022). Concentration of posterior model probabilities and normalized $L_0$ criteria. Bayesian Analysis, 17(2), 565-591.

Thank you for acknowledging the contribution of our work. Below we provide answers to address your specific concerns.

Weaknesses: Thank you for your thoughtful feedback regarding comparisons with existing frequentist approaches for model aggregation in conformal prediction. Based on your suggestions, we have revised our paper to include an expanded literature review of these frequentist approaches and a detailed discussion of the distinct contributions and scope of our Bayesian Model Averaging (BMA) framework within the conformal prediction literature.

We emphasize that frequentist methods often focus on effective set aggregation or selection but lack theoretical guarantees of optimal efficiency and valid statistical coverage. Our CBMA framework, on the other hand, directly addresses these gaps by providing theoretical guarantees for both optimal efficiency and valid statistical coverage in the resulting prediction sets under potential model misspecification.

For example, in the context of transductive conformal methods (also known as full conformal methods, which make full use of the data for prediction), Yang and Kuchibhotla (2024) provide selection rules to choose the smallest valid conformal prediction sets. However, their methods may suffer from coverage loss and lack theoretical guarantees for achieving statistical efficiency in the selected prediction set. In the context of inductive conformal methods (also known as split conformal methods, which require splitting the available data into training and calibration sets, reducing efficiency), prior works such as Vovk (2015), Linusson et al. (2017), Carlsson et al. (2014), Toccaceli and Gammerman (2019) propose methods that combine p-values generated by different models or data splits to create aggregated predictions. However, these combined p-values may not be well-calibrated, leading to prediction sets that do not accurately represent the intended coverage level. This lack of calibration undermines the reliability of the aggregated predictions.

Given these fundamental differences, direct comparisons between CBMA and frequentist model aggregation methods are not meaningful, as they lack rigorous guarantees of efficiency and coverage. For instance, we compared the lengths of CBMA aggregated intervals with those of conformal sets aggregated using the majority vote strategy (applying Corollary 4.1 from Gasparin and Ramdas (2024b) due to dependency among individual sets) in our real data example. The observed mean ratios of interval lengths constructed using majority vote scheme and our CBMA approach for $n = 50, 100, 150$ are $1.0765 (0.0253)$, $1.1196 (0.0151)$, and $1.1458 (0.0133)$, respectively, indicating that CBMA leads too shorter intervals. These results have been included in the revised paper's real data example section.

Question 1: We have revised Section 2.5 to provide a more detailed comparison of our method with existing approaches, highlighting their contributions, limitations, and how CBMA addresses these gaps. This expanded discussion clarifies the unique contributions of our work relative to the existing literature. Additionally, to address your concern about the weakness of our paper above, we have elaborated on why existing frequentist methods are not directly comparable to our approach, given the fundamental differences in objectives and guarantees.

Question 2: Following your suggestions, we have added the new experiments in Section 5.1 in the revised paper, with consistent experimental settings with other experiments ($\alpha = 0.2$, train-test split of $60-40$), which will improve the clarity and allow comparisons across experiments.

References:

Lars Carlsson, Martin Eklund, and Ulf Norinder. Aggregated conformal prediction. In Artificial Intelligence Applications and Innovations: AIAI 2014 Workshops: CoPA, MHDW, IIVC, and MT4BD, Rhodes, Greece, September 19-21, 2014. Proceedings 10, pp. 231–240. Springer, 2014.

Matteo Gasparin and Aaditya Ramdas. Conformal online model aggregation. arXiv preprint arXiv:2403.15527, 2024a.

Matteo Gasparin and Aaditya Ramdas. Merging uncertainty sets via majority vote. arXiv preprint arXiv:2401.09379, 2024b.

Henrik Linusson, Ulf Norinder, Henrik Bostr¨om, Ulf Johansson, and Tuve L¨ofstr¨om. On the cali- bration of aggregated conformal predictors. In Proceedings of the Sixth Workshop on Conformal and Probabilistic Prediction and Applications, volume 60 of Proceedings of Machine Learning Research, pp. 154–173. PMLR, 13–16 Jun 2017.

Vladimir Vovk. Cross-conformal predictors. Annals of Mathematics and Artificial Intelligence, 74: 9–28, 2015.

Yachong Yang and Arun Kumar Kuchibhotla. Selection and aggregation of conformal prediction sets. Journal of the American Statistical Association, pp. 1–13, 2024. doi: 10.1080/01621459. 2024.2344700.