We would like to thank the reviewer u4qk for recognizing the importance of drawing the parallel between OOD and CP. We address your remarks in the following.

• On uncertainty representation: Thank you for this interesting remark, we include it in the revised version of the manuscript and add that CP is also a method for uncertainty representation, supported by the reference you provided (thank you!). Still, we also leave that it is an uncertainty quantification technique, because it is broadly recognized as such by the community.
• We corrected the typo and corrected the statement about split CP being the simplest approach (now we say that it is one of the simplest approaches)
• About the work of Kaur [2]: After careful reading, we do not see connections with Kaur [2] other than the fact that they propose a new OOD detection score and a redefinition of OOD - which is related to one of the research fields of our work (OOD) but, in our opinion, not to our contributions. But we are curious about what you have in mind and would be glad to know about the link that you see if you see any!

We would like to thank reviewer 16UK for acknowledging the importance of providing probabilistic guarantees to evaluation metrics, especially for safety-critical applications. We appreciate that you grasped the importance of linking CP and OOD. We first address your main concern and then answer the question section.

Main concerns

1. The contributions are not limited to the corrected metrics

This criticism is supported by the observation that the technical foundations of our corrected metrics rely on Bates et al. (2022). This observation is true, and we acknowledge it several times in the manuscript. However, deriving corrected metrics is far from being the only contribution of this work. In addition to this contribution:

• We point out that the common practice of OOD and Anomaly detection enables the use of this theoretical work. Drawing this link is a contribution.
• We also use OOD scores to build new prediction intervals, which is not related to the work of Bates et al. (2022). It opens a whole avenue for nonconformity score crafting, which is central to CP performances.
• To us, perhaps the most important contribution is the message carried by the paper, supported by our several contributions : OOD and CP are closely related and should be used jointly. We believe that this message could have a significant impact on both fields, which are very dynamic and active research fields in AI.

We agree that each contribution taken independently would not be a sufficient contribution for a whole paper. But they form a coherent whole that advocates for further exploration of the link between CP and OOD, which is, in our opinion, a very important insight and could have a broad impact on future research in both fields.

2. We do not need extra validation (or calibration) data

We should have insisted more on that point because it is another advantage of our method: it does not require extra validation data.

Computing the conformal FPR only requires a correction on the estimated FPR that does not rely on a third calibration dataset. It is not like in CP where we need a calibration dataset to find a threshold based on nonconformity scores obtained on calibration data, that is subsequently used to provide CP confidence intervals. Here, there are no confidence prediction intervals; we only use CP theory to obtain probabilistic guarantees on the FPR. The steps are as follows:

• We compute the estimated FPR with the full validation data, as usual in OOD
• We apply the correction
• We compute the TPR using OOD validation data (as usual in OOD) but with a threshold that corresponds to the corrected FPR instead of the non-corrected FPR.

We added a remark in the main manuscript in Section 4.4 (Remark 4.1) to clarify this point.

Question

Classical CP scores rely on softmax (LAC, APS, RAPS), which can be related to prediction confidence. But it is also used as an OOD score [1]. In fact, many OOD scores are derived from softmax (energy [2], entropy [3], GEN [4]). More generally, as seen in OpenOOD's paper benchmark results, predictive uncertainty methods work well for OOD detection. It does not necessarily mean that one single metric can accurately capture both types of uncertainties, but it suggests that there may be a link between those two and that one is a good practical proxy for the other.

Still, OOD scores are not directly related to uncertainties, e.g., DKNN [6], which is, in fact, very competitive for OOD detection.

[1] Dan Hendrycks and Kevin Gimpel. A Baseline for Detecting Misclassified and Out-of-distribution Examples in Neural Networks.

[2] Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy-based Out-of-distribution Detection

[3] Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, Balaji Lakshminarayanan. Likelihood Ratios for Out-of-Distribution Detection

[4] Xixi Liu, Yaroslava Lochman, Christopher Zach. GEN: Pushing the Limits of Softmax-Based Out-of-Distribution Detection

Thank the authors for answering my questions and clarifying my doubts. I am happy to maintain my rating.

We would like to thank reviewer NLkM for his or her constructive feedback on the writing and the technical details of the paper. We first addressed the main concerns, which is the overlap between our work and the work of Bates et al. (2022) (seen in the weaknesses and the question section). Then, we address minor comments.

Main concern

The contributions are not limited to the corrected metrics

This criticism is supported by the observation that the technical foundations of our corrected metrics rely on Bates et al. (2022). This observation is true, and we acknowledge it several times in the manuscript. However, deriving corrected metrics is far from being the only contribution of this work. In addition to this contribution:

• We point out that the common practice of OOD and Anomaly detection enables the use of this theoretical work. Drawing this link is a contribution.
• We apply it to extensive OOD and Anomaly Detection benchmarks. You point out that the OOD baselines perform similarly to the vanilla metric. While you consider it as a weakness, we argue that it is a strength: it means that using corrected metrics, that provide safety guarantees, comes with almost no cost and could hence be used seamlessly in OOD. The conclusion is not the same for Anomaly Detection, which shows that current benchmarks are brittle when it comes to safety assessment, which is another strong insight.
• We also use OOD scores to build new prediction intervals, which is not related to the work of Bates et al. (2022). It opens a whole avenue for nonconformity score crafting, which is central to CP performances.
• To us, perhaps the most important contribution is the message carried by the paper, supported by our several contributions : OOD and CP are closely related and should be used jointly. We believe that this message could have a significant impact on both fields, which are very dynamic and active research fields in AI.

We agree that each contribution taken independently would not be a sufficient contribution for a whole paper. But they form a coherent whole that advocates for further exploration of the link between CP and OOD, which is, in our opinion, a very important insight and could have a broad impact on future research in both fields.

Minor remarks

• Thank you for pointing out the notation problem with $n_val$. We corrected it in the revised version of the manuscript.
• About $\tau$ and $t$: $\tau$ is used for threshold on the scores, and $t$ for threshold on the p value. $t \in [0,1]$ whereas $\tau \in \mathbb{R}$, so each of them has its own use.

Thank you for taking the time to read our response.

We regret that you deem our paper not worthy of publication due to the lack of methodological novelty, regardless of the potential impact of its message. We would like to make sure that you correctly caught that our contributions were not only about applying conformal prediction to OOD and AD, as you mentioned in your last message but also doing the converse and using OOD for CP.

Lastly, we would like to quote ICLR's reviewer guidelines: "Submissions bring value to the ICLR community when they convincingly demonstrate new, relevant, impactful knowledge (incl., empirical, theoretical, for practitioners, etc).". We admit that the paper does not bring many significant methodological novelties (though it brings several). Still, we would be disappointed that our message could not be heard in ICLR, despite its broad impact, just because of this reason. In our opinion, a methodologically simple paper with more potential impact is as worthy of publication as a paper of high technicality but with restricted application. But of course, it is only our opinion, and we undoubtedly respect yours

We would like to thank reviewer 41Qn for appreciating the writing of our paper and recognizing its importance and relevance to the ICLR community. We first answer your main concerns (also discussed in the general answer).

Main concerns

1. The contributions are not limited to the corrected metrics

This criticism is supported by the observation that the technical foundations of our corrected metrics rely on Bates et al. (2022). This observation is true, and we acknowledge it several times in the manuscript. However, deriving corrected metrics is far from being the only contribution of this work. In addition to this contribution:

• We point out that the common practice of OOD and Anomaly detection enables the use of this theoretical work. Drawing this link is a contribution.
• We apply it to extensive OOD and Anomaly Detection benchmarks. You point out that the OOD baselines perform similarly to the vanilla metric. While you consider it as a weakness, we argue that it is a strength: it means that using corrected metrics, that provide safety guarantees, comes with almost no cost and could hence be used seamlessly in OOD. The conclusion is not the same for Anomaly Detection, which shows that current benchmarks are brittle when it comes to safety assessment, which is another strong insight.
• We also use OOD scores to build new prediction intervals, which is not related to the work of Bates et al. (2022). It opens a whole avenue for nonconformity score crafting, which is central to CP performances.
• To us, perhaps the most important contribution is the message carried by the paper, supported by our several contributions : OOD and CP are closely related and should be used jointly. We believe that this message could have a significant impact on both fields, which are very dynamic and active research fields in AI.

We agree that each contribution taken independently would not be a sufficient contribution for a whole paper. But they form a coherent whole that advocates for further exploration of the link between CP and OOD, which is, in our opinion, a very important insight and could have a broad impact on future research in both fields.

2. We do not need extra validation (or calibration) data

We should have insisted more on that point because it is another advantage of our method: it does not require extra validation data.

Computing the conformal FPR only requires a correction on the estimated FPR that does not rely on a third calibration dataset. It is not like in CP where we need a calibration dataset to find a threshold based on nonconformity scores obtained on calibration data, that is subsequently used to provide CP confidence intervals. Here, there are no confidence prediction intervals; we only use CP theory to obtain probabilistic guarantees on the FPR. The steps are as follows:

• We compute the estimated FPR with the full validation data, as usual in OOD
• We apply the correction
• We compute the TPR using OOD validation data (as usual in OOD) but with a threshold that corresponds to the corrected FPR instead of the non-corrected FPR.

We added a remark in the main manuscript in Section 4.4 (Remark 4.1) to clarify this point.

Questions

1. In Figure 2, we show that the AUROC is overestimated using classical evaluation of CIFAR10 vs SVHN. The experimental results of Section 4 contribute to safer benchmarks because they limit the risk of the AUROC and FPR being overestimated, which is highly undesirable for safety-critical applications. Certification processes that ensure safety levels for any safety critical systems seek guarantees that the estimations are conservative with controlled probabilities. This is not provided by classical metrics.
2. The guarantees stemming from conformal prediction are automatically preserved in whatever score we use, even if it is an OOD score. This is one of the strengths of the approach: we highlight that every new OOD score is a potential nonconformity score for CP.

Thank you very much for increasing your score, though not up to acceptance. We regret that you deem our paper not worthy of publication because you judge the technical contribution as not sufficient, regardless of the potential impact of its message.

Lastly, we would like to quote ICLR's reviewer guidelines: "Submissions bring value to the ICLR community when they convincingly demonstrate new, relevant, impactful knowledge (incl., empirical, theoretical, for practitioners, etc).". We admit that our work is not highly technical (though it required significant empirical and implementation efforts to run all the benchmarks and to adapt OOD scores to CP, respectively). Still, we would be disappointed that our message could not be heard in ICLR, despite its broad impact, just because of this reason. In our opinion, a technically simple paper with more potential impact is as worthy of publication as a paper of high technicality but with restricted application. But of course, it is only our opinion, and we undoubtedly respect yours.