On Temperature Scaling and Conformal Prediction of Deep Classifiers

2. "The somewhat expected nature of the finding that TS might impact set size negatively."

Please note that our empirical and theoretical study shows that TS can also affect the prediction set size positively (at the price of degradation in conditional coverage) and not only negatively.

We believe that our discoveries on the effect of TS on CP methods are not intuitive.

When initiating our research, we hypothesized that CP algorithms would overall benefit from a well-calibrated model, as it seemed reasonable to assume that softmax outputs aligned more closely with true correctness probabilities would enhance performance. Presumably, this has also been the intuition of all the works (cited in our paper) that applied TS calibration before applying CP without any examination of the effect of this procedure on the CP methods (Angelopoulos et al., 2021; Lu et al., 2022; Gibbs et al., 2023; Lu et al., 2023).

However, upon further investigation, we realized that the behavior for APS and RAPS differs from LAC, and that the primary factor that affects APS and RAPS is not the calibration level itself but rather the temperature value, $T$.

Specifically, as presented in our paper, we discovered that when $T<1$ then TS positively affects the prediction set size of APS and RAPS, but at the price of negative effect on their class-conditional coverage. And vise versa, for $1< T < T_{critical}$ (negative effect on prediction set size but positive effect on conditional coverage). All the more so, the phenomenon where there exists $T_{critical}$ such that for TS with $T>T_{critical}$ the trends (degradation/improvement) in the prediction set size and conditional coverage are swapped is also very non-intuitive.

We thank the reviewer for their overall positive feedback and are pleased that they recognize the importance of our work and the novel insights we provide regarding the effect of TS on CP. Below we provide a detailed response to the reviewer's comments.

1. "The limited scope of the paper to classification tasks, as regression tasks do not typically present this issue."

We believe that exploring and understanding the interplay between calibration and CP in regression is a great idea for future research, separated from this paper. Below we present justifications for our focus on classification:

• Importance of classification tasks. The classification task is extremely prevalent in practice and is an impactful area of research, with countless significant studies dedicated solely to classification. In fact, many seminal machine learning ideas [1],[2],[3] were originally presented in the context of classification and only later applied to other tasks, such as regression, in separated works.
  Our work focuses on two post-hoc uncertainty quantification methods: TS calibration and CP. Seminal studies in these topics, such as [4],[5],[6], have also been dedicated exclusively to classification, underscoring the relevance and importance of our focus.

• Addressing regression tasks in addition to classification tasks is beyond the scope of this paper. Regression constitutes a fundamentally different task than classification, and as such, uncertainty quantification methods are very different in regression and classification. For example, CP outputs a prediction set (of discrete labels) in classification but a continuous interval in regression.
  Our study delivers a comprehensive investigation into classification tasks, featuring extensive empirical analysis and a rigorous theoretical exploration of our findings. Tackling regression tasks would require a fundamentally different set of empirical experiments and theoretical approaches. Thus, extending our work to encompass regression tasks would be beyond the feasible scope of a single paper.

We would like to emphasize again that we fully acknowledge the significance of regression problems. We view the investigation of the interplay between calibration and CP in regression as a promising direction for future research, as we mention in the conclusion of the revised paper in line 538.

============================

[1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems 25 (2012).

[2] Sergey Ioffe and Christian Szegedy. "Batch normalization: Accelerating deep network training by reducing internal covariate shift." arXiv preprint arXiv:1502.03167, 2015.

[3] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Deep residual learning for image recognition." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.

[4] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. "On calibration of modern neural networks." In International conference on machine learning, pp. 1321–1330. PMLR, 2017.

[5] Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan, and Jitendra Malik. "Uncertainty sets for image classifiers using conformal prediction." In International Conference on Learning Representations, 2021.

[6] Yaniv Romano, Matteo Sesia, and Emmanuel Candes. "Classification with valid and adaptive coverage." Advances in Neural Information Processing Systems 33 (2020): 3581-3591.

Dear Reviewer yXD1,

We would greatly appreciate receiving your feedback on our responses before the discussion period ends (in less than two days).
Have we addressed your comments?

Thank you again for the review.

We sincerely thank the reviewer for this detailed review. Below we provide our point-to-point response to each of their concerns. We hope that our detailed response will lead to reconsidering the score.

1. On the link between Eq. (3) and Eq. (5)

We sincerely thank the reviewer for this comment. Indeed, there was a missing part in the derivation in the original version which relates $g(z^q,T,M)$ (associated with Eq. 5) to $g(z^q,T,L^q)$ (associated with $\hat{q}-\hat{q}_T$ in Eq. 3), where by $L^q$ we denote the rank of the true label of the quantile sample $z^q$.

We empirically observed that $|g(z^q,T,M) - g(z^q,T,L^q)|$ is negligible in our experiments (where $\Delta z^q >> 1$). Moreover, we also managed to show theoretically that $|g(z^q,T,M) - g(z^q,T,L^q)|$ decays exponentially with $\Delta z^q$.

Accordingly, in the revised version, before Eq. 5, we added this clarification which refers the reader to Proposition A.6 in the appendix, where we formally show (under minor assumption which is fully empirically justified) that $|g(z,T,M) - g(z,T,L)| < \frac{(C-1)\exp{(-\Delta z)}}{(C-1)\exp{(-\Delta z)} + 1} \quad \text{If} \quad 0 < T < 1$ $|g(z,T,M) - g(z,T,L)| < \frac{(C-1)\exp{(-\Delta z / T)}}{(C-1)\exp{(-\Delta z / T)} + 1} \quad \text{If} \quad T > 1$

This formally implies that $|g(z^q,T,M) - g(z^q,T,L^q)|$ is small since $\Delta z^q >> 1$.

To conclude, our proof for the bound of $|g(z^q,T,M) - g(z^q,T,L^q)|$ as well as the empirical values presented in Appendix A.1, link Eq. (3) and Eq. (5). Furthermore, the relevance of our theoretical analysis is further reinforced by the fact that the analysis of Eq. (5) provides fine-grained explanations for the behavior observed in the experiments (as we explain in the response to the comments below --- the theory indeed agrees with the experiments).

Again, we are grateful to the reviewer for pointing out this issue, and as mentioned, we addressed this in the revised version in lines 428–432 in the main paper and in Appendix A.1.

 
2(a). On the assumptions in Theorem 4.4 - line 458 in the original version

It seems there might have been some misunderstanding regarding the sentence in line 458 in the original version (now associated with lines 451-452 in the revision).

Theorem 4.4 indeed tells that if $\Delta z > b(T)$, then increasing $z_1$ leads to an increase in the prediction set size. However, contrary to the reviewer's comment, in line 458 there was no additional constantness assumption on $\Delta z$.

The complete sentence in line 458 of the original version, which the reviewer refers to, is:
"indeed, the condition on $\Delta z$ is preserved as $z_1$ increases."

We do not require that $\Delta z$ is preserved or constant if $z_1$ increases (which cannot be true). On the contrary, for $T>1$, we explain that if $z_1$ increases then $\Delta z$ increases, and therefore $\Delta z > b(T)$ continues to hold, i.e., the condition on $\Delta z$ in the theorem still holds.

Following this comment, in the revision we rephrased the sentence for better clarity (lines 451-452 in the revised version):
"indeed, when $z_1$ increases then $\Delta z$ increases, and thus the inequality $\Delta z > b_{T>1}(T)$ remains satisfied."

 
2(b). On the assumptions in Theorem 4.4 - larger $\Delta z$ for $z^q$

Regarding the justification of "$z_q$ has a larger dominant entry than typical $z$": As stated in lines 418-431 of the original version (406-420 in the revised version), the score of the quantile sample $z_q$ exceeds the scores of $(1 - \alpha)\%$ (e.g. 90%) of the other samples. Figure 3 (Figure 4 in the original version with a minor fix) illustrates a strong correlation between the score and $\Delta z$. Since $z^q$ has higher score than the median score, associated with some $z'$, this empirically supports the condition that it has larger $\Delta z$. In numbers, in Figure 3, the quantile sample has $\Delta z^q \approx 11$ while the median sample has $\Delta z' \approx 8$ and larger $\Delta z = z_{(1)}-z_{(2)}$ implies more dominant highest entry. The difference is even enhanced after the softmax due to the relation $\pi_i \propto \exp{z_i}$.

3. On the agreement of the theoretical bounds and the empirical results

We believe there may have been a misunderstanding of the reviewer regarding the content of the paragraph in lines 463-470 in the original version (lines 458-465 in the revised version).

We demonstrate that our theory not only provides intuition for the phenomenon but also offers reasonable bounds that explain the empirical observations for a "typical" sample in the CIFAR100-Resnet50 setting. For this typical sample, which we interpret as the sample with the median score, we have $\Delta z \approx 8$. Based on Theorem 4.4 (when we plug in it $\Delta z \approx 8$ and $C=100$), the conditions on $\Delta z$ of the theorem hold for temperatures in the ranges $(0, 0.831)$ and $(1.25, 4.81)$.

The optimal calibration temperature for CIFAR100-Resnet50 is $T^* = 1.524$. The reviewer incorrectly referred to it as the critical temperature (which we defined differently - $T_{critical}$ is the temperature where the trend in the curve of the prediction set size vs. $T$ changes from increase to decrease).

$T^* = 1.524$ indeed falls within the range where the theorem is valid. The theorem predicts that for this temperature the prediction set size of a typical (median-score) sample increases. This is aligned with the increase in the mean prediction set size that we see in Table 1 for APS for CIFAR-100, ResNet50 after performing TS calibration (which resulted in using $T^* = 1.524$).

To conclude, there is no inconsistency here.

 
4. Regarding the proposed approach in Section 5

In continuation with our response to the previous comment, it seems that reviewer confused between $T^*$ and $T_{critical}$. Accordingly, their claim: "As discussed in Weaknesses[3], the theorem may fail..." is not justified.

The proposed approach in Section 5 suggests to apply TS separately for calibration and for CP, and explains how the samples that are used for calibration can be utilized to estimate the curves of the CP method (AvgSize and TopCovGap vs. T) as demonstrated in Appendix C. As we explain in Section 5, the range of $T$ for exploration can be bounded by $T_{critical}$ that can be approximated by our theory. Again, this is not the temperature $T^*$ that is optimal for calibration but rather the intersection of the two functions $\frac{T}{T-1}\ln(4T)$ and $\frac{T-1}{T+1}\ln(4T(C-1)^2)$ in the $T>1$ branch in Eq. 6.

Since our guideline allows the user to set a temperature given their preferences between AvgSize and TopCovGap, it has practical significance.

Regarding the Mondrian conformal prediction [1], this CP method is based on obtaining CP thresholds separately per class. We observed that our approach outperforms it in our experiments, which is aligned with its inferior performance reported in previous paper when the number of samples used for calibrating CP per class is small [2].

Specifically, in our experiments, we consider CIFAR-100 that has $C=100$ classes and CP set (used to calibrate the CP) of size up to 2000, and ImageNet that has $C=1000$ classes and CP set of size up to 10000. This means that approximately we have only 20 samples per class to calibrate CP for CIFAR-100 and 10 samples per class to calibrate CP for ImageNet.

Our metric TopCovGap measures worst-case class-coverage violation. For a user that prioritizes class-conditional coverage, our detailed research and guidelines in Section 5 suggest to use TS with $T=T_{critical}$, where AvgSize is the high, but by the trade-off TopCovGap is low.
The properties of RAPS with such TS outperforms Mondrian CP in both TopCovGap and AvgSize.

We thank the reviewer for mentioning the Mondrian CP. We discuss and compare our approach to it in the revision in lines 523-526 and in Appendix D. These results, which show that our approach outperforms it, further strengthen the practical significance of our contribution.

=========

[1] Vladimir Vovk. "Conditional validity of inductive conformal predictors." In Asian Conference on Machine Learning, 2012.

[2] Ding, Tiffany, et al. "Class-conditional conformal prediction with many classes." Advances in Neural Information Processing Systems, 2024.

Minor comment 1: "The paper's analysis is limited to Temperature Scaling"

We will raise several justifications of our study on TS confidence calibration method:

• Popularity and efficiency of Temperature Scaling. As stated in lines 44-48, "Guo et al. (2017) demonstrated the usefulness of a simple Temperature Scaling (TS) procedure (a single parameter variant of Platt scaling (Platt et al., 1999)). Since then, TS calibration has gained massive popularity (Liang et al., 2018; Ji et al., 2019; Wang et al., 2021; Frenkel & Goldberger, 2021; Ding et al., 2021; Wei et al., 2022). Thus studying it is of high significance."
• Utilization of Temperature Scaling as a pre-process step to Conformal Prediction. As we mentioned in line 60, the previous papers (Angelopoulos et al., 2021; Lu et al., 2022; Gibbs et al., 2023; Lu et al., 2023) that apply initial calibration before applying CP methods, use TS calibration algorithm rather than any other calibration method.
• The ability to develop a theory on Temperature Scaling. The simplicity of the TS procedure enabled us to develop a theoretical framework for its effect on CP, which is a notable achievement given the complexity of the problem.
• Temperature Scaling can be used beyond calibration. Since TS is based on a single parameter (the temperature), as discussed in Section 5 based on our discoveries, it paves the way for practitioners to use TS beyond the value associated with calibration in order to trade prediction set sizes and conditional coverage of adaptive CP methods (APS and RAPS) according to their specific needs.

Therefore, it is fair to focus on TS in this paper and defer the study of other calibration methods to future research.

 
Minor comment 2: Clarity of the mathematical proofs

Following this comment, in the revised version we added explanations to steps in the proofs.

Regarding the specific question of the reviewer on lines 750-755 in the original version (proof of Theorem A.1), according to the lemma in Theorem A.1, $\forall z_i \geq z_j, T \geq \tilde{T} \geq 0$ The following holds:

$

\exp{(z_i/\tilde{T})} \cdot \exp{(z_j/T)} \geq \exp{(z_i/T)} \cdot \exp{(z_j/\tilde{T})}

$

Note that we denoted the sets of the indices $I = [1, 2, \ldots, L]$ and $J = [L+1, L+2, \ldots, C]$. In the theorem, we deal with a sorted logits vector, and therefore, as stated in the original version: "Because $z$ is sorted, $\forall i \in I , j \in J$ we have $z_{i}>z_{j}$." So, we have the above inequality for each $i \in I , j \in J$. To obtain the inequality in line 750 in the original version we merely sum the inequality over all $i \in I , j \in J$ combinations: $\sum_{i=1}^L \sum_{j=L+1}^C \exp{(z_i/\tilde{T})} \cdot \exp{(z_j/T)} \geq \sum_{i=1}^L \sum_{j=L+1}^C \exp{(z_i/T)} \cdot \exp{(z_j/\tilde{T})}$ To get to line 755 in the original version, we separated the summation of the indexes $i \in I, j \in J$: $\sum_{i=1}^{L}\exp(z_{i}/\tilde{T}) \cdot \sum_{j=L+1}^{C}\exp(z_{j}/T) \geq \sum_{i=1}^{L}\exp(z_{i}/T) \cdot \sum_{j=L+1}^{C}\exp(z_{j}/\tilde{T}).$ In the revision, we stated verbally the summation and separation of the sums.

 
Minor comment 3: typo in Section 5

Thank you for bringing this to our attention. We have carefully reviewed the paper for language and typographical errors and made corrections where necessary.

 
Question 1: Computing $T_{critical}$

In our paper (and has been emphasized in the revision in lines 477-480, 491-492), $T_{critical}$ is the temperature where the trend of the prediction set size, as $T$ increases, changes from increasing to decreasing. Our theory allows to approximate it by computing the intersection of the two functions $\frac{T}{T-1}\ln(4T)$ and $\frac{T-1}{T+1}\ln(4T(C-1)^2)$ in the $T>1$ branch in Eq. 6. This intersection has no analytical solution but can be easily computed numerically. We now mention this in lines 477-480 in the revised version.

Dear Reviewer eNPR,

We would greatly appreciate receiving your feedback on our responses before the discussion period ends (in less than two days).
Have we addressed your comments?

Thank you again for the review.

We thank reviewer eNPR for their comment and glad to hear we addressed their concerns regarding our theoretical analysis. We respectfully disagree on their claims on the “limited contribution of the work”. Before we respond to their comment point-to-point, we would like to emphasize that we addressed all the main concerns raised by the reviewer in the first round. In this round, the reviewer focuses on one minor comment (minor comment number 1 from their first post) and presents new claims on the contribution of the work. Below we rebut each of their claims.

The scope of the main analysis is limited in the temperature scaling in classification

Regarding the focus on Classification.

As we commented in the first round, classification constitutes as an important task, and we justify our focus on it with the following:

• Importance of classification tasks.

The classification task is extremely prevalent in practice and is an impactful area of research, with countless significant studies dedicated solely to classification. In fact, many seminal machine learning ideas [1],[2],[3] were originally presented in the context of classification and only later applied to other tasks, such as regression, in separated works.

Our work focuses on two post-hoc uncertainty quantification methods: TS calibration and CP. Seminal studies in these topics, such as [4],[5],[6], have also been dedicated exclusively to classification, underscoring the relevance and importance of our focus.

• Addressing other tasks in addition to classification tasks is beyond the scope of this paper.

The only other task that may be as prevalent in machine learning as classification is regression. Let us explain why it is not reasonable to include it together with classification in our paper.

Regression constitutes a fundamentally different task than classification, and as such, uncertainty quantification methods are very different in regression and classification. For example, CP outputs a prediction set (of discrete labels) in classification but a continuous interval in regression.

Our study delivers a comprehensive investigation into classification tasks, featuring extensive empirical analysis and a rigorous theoretical exploration of our findings. Tackling regression tasks would require a fundamentally different set of empirical experiments and theoretical approaches. Thus, extending our work to encompass regression tasks would be beyond the feasible scope of a single paper.

---

Note that Reviewer yXD1 acknowledged the importance of classification task in the second round and noted that this does not influence their evaluation.

[1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems 25 (2012).

[2] Sergey Ioffe and Christian Szegedy. "Batch normalization: Accelerating deep network training by reducing internal covariate shift." arXiv preprint arXiv:1502.03167, 2015.

[3] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Deep residual learning for image recognition." In Proceedings of the IEEE conference on computer vision and pattern recognition,
pp. 770–778, 2016.

[4] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. "On calibration of modern neural networks." In International conference on machine learning, pp. 1321–1330. PMLR, 2017.

[5] Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan, and Jitendra Malik. "Uncertainty sets for image classifiers using conformal prediction." In International Conference on Learning Representations, 2021.

[6] Yaniv Romano, Matteo Sesia, and Emmanuel Candes. "Classification with valid and adaptive coverage." Advances in Neural Information Processing Systems 33 (2020): 3581-3591.

Regarding the focus on Temperature Scaling.

As we answered in the first round to Minor comment 1, TS calibration constitutes as an important confidence calibration method, and we provided several justifications for our focus on it in this paper.

We are grateful to the reviewer for the positive review, which acknowledges the importance of our work, and appreciates our empirical experiments and theoretical analysis. Below is our response to the comments.

• Values of the theorems

Indeed, the core goal of our theoretical study is to provide mathematical reasoning for the surprising phenomena that we discovered empirically. We believe that the "why" question is important when the observations are non-intuitive. Yet, at the same time, practitioners can utilize Theorem 4.4 to approximate the temperature $T_{critical}$ where the monotonicity in the CP properties breaks, and utilize it when defining a range for tuning $T$ to trade between prediction set sizes and class-conditional coverage according to their specific requirements, as described in Section 5. For example, if class-conditional coverage is a key priority for the practitioner, selecting $T = T_{critical}$ might be an appropriate choice. And indeed, in Appendix D in the revised version we show the advantages of this approach over an existing approach.

• Typo/English language check

We have carefully reviewed the paper for language and typographical errors and made corrections where necessary.

We sincerely thank all the reviewers for their time and efforts and their valuable feedback.

We would like to reiterate the key contributions of our study.

• We discovered surprising empirical phenomena regarding the impact of TS (beyond calibration) on CP, revealing non-monotonic trade-off effects that depend on the temperature.

• We introduce a novel theoretical framework that explains our findings and formalizes the impact on prediction set sizes, which can be useful for future research.

• We provide practical insights, introducing a novel approach for trading prediction set sizes and conditional coverage by tuning a single temperature parameter (unrelated to the one used for calibration).

• When prioritizing prediction set sizes or class-conditional coverage, TS with our theoretically and empirically backed temperature values outperforms existing adaptive CP techniques.

 
These contributions have been recognized by the majority of the reviewers.

Reviewer 3FBk stated that "the paper is strong" and that they were "impressed and surprised by the insights in this paper". They praised also our experiments as "painstakingly detailed", and recognized our theoretical analysis as "useful and correct" and "might be useful to others in the field."

Reviewer yXD1 stated that "the paper focuses on an interesting question" and that they "find this paper offers valuable experiments and intriguing insights".

All the three reviewers recognized the presentation, stating that the paper is “well written” and “clear”.

Reviewer eNPR is the only reviewer with negative recommendation. In the rebuttal we fully addresses all their comments. Notably, out of the weaknesses raised, only one was a valid concern, which we have fully addressed (see the first point in our previous general response). We also strengthened our practical contribution (see the second point in our previous general response).
When responding to our rebuttal, Reviewer eNPR based their negative score on claims about the contribution of our work, which we have rebutted above, and argued that studying TS does not suffice because:
"previous works in conformal prediction only use the temperature scaling as a small trick and do not promote it as an effective tool to improve the efficiency or conditional coverage."
However, the fact that these previous works missed/overlooked the phenomenon that TS significantly affects the performance of adaptive CP in a non-intuitive trade-off manner, and that we discovered it, is a strength of our work, not a weakness.

In the rebuttal (and in the paper itself) we also detailed the reasons that justify the focus of the paper on TS, which include:

• Enormous popularity and efficiency of TS calibration
• Existing CP works did not use any other calibration method
• TS can be used beyond calibration
• The ability to develop a theory on TS

More details for each of these points can be found below in the response to Reviewer eNPR. Therefore, we believe that it is fair to focus on TS in this paper (which is already full of content) and defer the study of other calibration methods to future research.

To conclude,
given all the above: 1) significant contributions (from deep empirical examination and discoveries to rigorous theoretical understanding and practical implications), 2) justification of focusing on TS, and 3) detailed rebuttal (which addresses all the points raised by the single negative reviewer), we sincerely hope that our paper will be accepted.

Dear authors,

Rescaling the entries of a vector (with the same scalar) obviously does not change the order of its elements, so does not change the argmax used to attribute the classes. Here, the term "accuracy" obviously refers to "accuracy of the distribution of the score" (that is how well the model' predicted probabilities reflects the true likelihood) which is an output of the model itself (predicted classes, predicted confidence).

Sorry, if the wording was ambiguous but Here, there is no reclassification involved anyway, so the context is very clear.

Temperature scaling recalibrates the model outputs i.e. aligns frequency of correct prediction with the predicted confidence. Since the conformity scores of LAC, APS, RAPS are directly based on these (potentially recalibrated) confidence values, any change in the calibration is naturally ("expected" to) reflected in the CP outcomes. This was a point discussed during the review process and reported in my meta-review.

Regards,

AC

Hi authors,

I'd like to clarify that my concerns are not addressed in the rebuttal and I hope this reply can help you in the next version. One of the main concerns is the practical value of the theoretical results (Section 5). The theory shows how to find the worst T that leads to the largest sets. However, it does not show how to find the sweet spot (best T) of the tradeoff between efficiency and conditional coverage. In Section 5, the authors claim that one can select different T according to the target (size/covGap/ECE):

1. For the smallest size, we have to conduct a grid search to find the optimal T (should be in (0,1], instead of (1, $\infty$)), which is irrelevant to the theory.
2. For the smallest CovGap, the authors suggest using the worst T for the size, which cannot guarantee the best CovGap but the largest Size. This is not convincing as a meaningful guideline in practice.

Therefore, I believe this paper should be improved in the next version. About the theory, the non-monotonic trend in (1, $\infty$) is interesting but might be meaningless as it does not have practical value. Finally, the writing should also be improved as it is hard for readers to understand the notation and logic (such as using the worst T ($T_{critical}$) to prioritize class-conditional coverage).

Best wishes,

Reviewer eNPR

We thank Reviewer eNPR for their engagement and new comments (which do not appear below). 

We encourage visitors of the page to read the previous discussion where we rebutted the reviewer's earlier comments.

Below are our responses addressing each of the new comments.

First, note that CovGap (the marginal coverage) is ensured, so we assume that the reviewer means TopCovGap, which reflects class-conditional coverage.

Second, our work includes both empirical (discoveries, practical guidance) and theoretical contributions.
We believe that equipping surprising, nonintuitive empirical discoveries with mathematical theory that explains them is very important in exact sciences.

Since there is a trade-off between AvgSize and TopCovGap for APS/RAPS, there is no single value of $T$ that is optimal in terms of both. (The term "sweet spot" used by Reviewer eNPR is not well defined).

In Section 5 and Appendix C we present a practical way that each user can set $T$ based on their personally preferred balance between AvgSize and TopCovGap.

1. If one cares only about AvgSize and wants to use APS/RAPS, the $0<T<1$ branch of our theory suggests that he picks the smallest $T$ (which is aligned with the empirical observations, up to numerical issues that arise when approaching 0).  Furthermore, grid search has negligible complexity compared to training DNNs (in the considered settings, it takes no more than a few minutes).
2. Section 5 and Appendix C provide meaningful guidance that goes beyond the theory and allows to directly set a value of $T$ that is approximately optimal for TopCovGap. We emphasize and demonstrate this in a newer version. Typically, due to the trade-off that we discovered, this value is close to the value of $T$ where AvgSize is the largest.

In a newer version we further clarify the notation and discussion on the critical temperature. Yet, this is a minor issue that did not justify rejecting the paper. Overall, the reviewers complimented/gave a good grade to our writing/presentation (including Reviewer eNPR in their first review).

To conclude, Reviewer eNPR's new comments have also been addressed, and the suggested modifications are minor. As such, in our opinion, they do not justify the disappointing decision. We hope for a better experience in the future.