Delving into Temperature Scaling for Adaptive Conformal Prediction

1. ConfTS preserves uncertainty [W1,2]:

Thank you for raising this important concern about uncertainty quantification. We would like to address this from two key aspects:

• Our method does not conflict with confidence calibration, as it only replaces the temperature value. During inference, one may use different temperature values according to the objective, whether for improved calibration performance or efficient prediction sets.
• ConfTS maintains the adaptiveness property of adaptive conformal prediction. In particular, as shown in Figure 3 (a)&(b), our method produces smaller prediction sets for easy examples and larger sets for hard ones. This indicates that while ConfTS optimizes for efficiency, it still preserves the instance uncertainty where ambiguous cases receive larger prediction sets.

These properties indicate that ConfTS does not compromise uncertainty quantification.

2. The performance of ConfTS when the oracle classifier is available [W3]:

Thank you for this insightful suggestion. We designed a synthetic experiment with controlled uncertainty conditions to analyze this question. In the experiment, we consider a 10-class classification problem with 200-dimensional data and implement an oracle classifier that knows the true data generation process. We control the inherent uncertainty by adding Gaussian noise to the logits, where higher noise levels represent more inherent uncertainty in the classification task. We employ APS to generate prediction sets. To ensure robustness, each experiment is repeated 100 times, and we report the average results. The results demonstrate the relationship between task uncertainty and ConfTS's effectiveness.

The results show that ConfTS consistently reduces the average prediction set size across all noise settings. However, ConfTS increases the CSCV as we add noise to the logits. This suggests that ConfTS is particularly effective in improving conditional coverage when the classification task is more deterministic. We add this experiment in Appendix M.

3. Why minimizing the efficiency gap preserves adaptivity [Q1,2]:

There might be some misunderstandings. We clarify that adaptivity, as defined in prior work [1,2,3], requires prediction sets to communicate instance-wise uncertainty such that easy examples obtain smaller sets than hard ones. Notably, adaptivity is a property required by prediction sets, instead of that for non-conformity scores. Therefore, the distance between non-conformity scores could be unrelated to the concept of adaptivity.

In our experiments, we employ the label order (the position of the true label y in the sorted softmax probabilities π(x)) to reflect instance difficulty. This is because when a model is more confident about its prediction, the true label tends to have a higher softmax probability and thus a lower label order, and vice versa. Therefore, we can effectively examine how prediction set sizes vary with instance difficulty by partitioning samples in label order (1, 2-3, 4-6, 7-10, 11-100, 101-1000) following [2]. As demonstrated in Figure 3a and 3b, ConfTS maintains this adaptiveness property: examples with lower label order (easier samples) consistently obtain smaller prediction sets. This confirms that our method maintains adaptiveness while improving prediction efficiency.

4. The performance of ConfTS on conditional coverage [Q3]:

See general response.

References

[1] Yaniv Romano, et al. Classification with valid and adaptive coverage. NIPS 2020.

[2] Anastasios Nikolas Angelopoulos, et al. Uncertainty sets for image classifiers using conformal prediction. ICLR 2021.

[3] Nabeel Seedat, et al. Improving adaptive conformal prediction using self-supervised learning. ICML 2023.

Thank you for the patient discussion. In the following, we respond to the concerns point by point. 1. Experiments with an oracle classifier Thank you for the suggestion. In Section 5 of the revised paper, we add the statement in the analysis of conditional coverage.

2. The effect of the proposed loss There might be some misunderstanding. Firstly, we have a general assumption that the score function is not a constant function. Then, our loss function is $\mathcal{L}(x,y,t) = (\tau(t)-\mathcal{S}(x,y,t))^2$, where $\mathcal{S}(x,y,t)$ denotes the scores of examples with their true labels (not all labels) and $\tau$ is the $1-\alpha$ quantile of those scores. Thus, the optimization of the proposed loss pushes the scores of ground-truth labels (not all labels) to be close to the threshold $\tau$ for all examples. Ideally, the optimal case is that all the scores of true labels have the same value (as $\tau$), which leads to prediction sets that exactly cover the true labels (the smallest prediction set that contains the ground-truth label).

3. The limitation of the theoretical analysis Thank you for raising this concern. In the presence of noise, we can decompose the ConfTS loss $|S'-\tau'|$ into two components: the clean ConfTS loss $|S-\tau|$ and the inherent noise term (the distance between noisy and clean scores). As the noise level increases, the inherent noise term becomes dominant, making the upper bound on conditional coverage less tight. Consequently, while our theoretical results hold in general, the practical benefits are more pronounced in deterministic settings where the noise term is minimal.

6. The statistical and practical significance of the ConfTS improvement over ConfTr [Q6]:

Thank you for this great suggestion. Let us address both statistical and practical significance.

Statistical significance: In Table 3, we provide the results of two-sample t-test at a 0.1 confidence level. The p-values for our comparisons are presented in the table below. The small p-values (<0.1) across all tests indicate that these improvements are statistically significant.

Practical significance: As established in our core contribution, we demonstrate that temperature values optimized for confidence calibration are suboptimal for adaptive conformal prediction, and we propose a method to identify the optimal temperature specifically for this task. The consistent improvements observed in the experiment of ConfTr further validate our conclusion in the training method. Moreover, as demonstrated in previous work [3], the reduction in the prediction set size has practical significance, as smaller prediction sets are more informative to enable accurate human decision making.

7. The performance of ConfTS on conditional coverage [Q7]:

See general response.

8. Minor comments:

We fix these issues accordingly. It is a common practice to put the related work at the end of the manuscript so that readers can quickly touch the definition of problem setting.

References

[1] Sadinle M, et al. Least ambiguous set-valued classifiers with bounded error levels. JASA 2019.

[2] Guo C, et al. On calibration of modern neural networks. ICML 2017.

[3] Jesse C, et al. Conformal Prediction Sets Improve Human Decision Making. ICML 2024.

1. Choice of temperature scaling [Q1]:

There might be some misunderstanding. We clarify that this work is the first to study temperature scaling in conformal prediction, instead of applying conformal prediction for temperature scaling. Another point we need to clarify is the results of Table 1, where we show that all the calibration methods lead to large prediction sets in adaptive conformal prediction. This observation motivates us to investigate how high-confidence predictions affect conformal prediction.

Among these methods, we pick temperature scaling as it is the simplest method to adjust the confidence level with only a temperature parameter $T$. This enables us to provide a thorough analysis with theoretical and empirical results, revealing the relationship between confidence calibration and conformal prediction.

In addition, we also provide new results to show the advantage of ConfTS over the variant of vector scaling (ConfVS) trained by our loss. The results below show that temperature scaling is a better choice for this task as a simple and effective solution, while ConfVS can also improve efficiency.

2. Clarification on the performance of ConfTS [Q2]:

We clarify that ConfTS is designed as a complementary method rather than a competing alternative to existing approaches. The core contribution of our work is the observation that temperature values optimized for confidence calibration are not optimal for adaptive conformal prediction. We then propose a method to identify the optimal temperature specifically for adaptive conformal prediction. Our experimental results demonstrate that ConfTS can enhance various existing methods, including post-hoc and training approaches. We would greatly appreciate it if you could provide any suggestions for additional baseline methods or specific real-world datasets.

3. Using the same temperature for all classes [Q3]:

Thank you for this insightful suggestion about class-wise temperature scaling. In this work, we adopt the standard temperature scaling in the analysis and method design, for simplification and broad application (class-wise temperature scaling is not a common practice currently). While we acknowledge that class-specific temperature scaling could be beneficial, we employ a single temperature parameter in our method given several practical tradeoffs:

• Optimizing class-specific temperatures would require significantly more validation data to obtain reliable estimates for each class, which is particularly challenging for classes with fewer samples in imbalanced datasets.
• The optimization process for multiple temperature parameters would substantially increase training time, potentially limiting the method's practical applicability.

We believe class-wise temperature scaling might be an interesting direction for future work, but it does not affect the contribution of this work, as demonstrated in the reply to the last concern.

4. Interpretation of Theorem 3.3 [Q4]:

Thank you for pointing out this important question. Theorem 3.3 serves primarily as theoretical motivation for our method. In particular, it demonstrates that high-confidence predictions produced by a small temperature could lead to efficient prediction sets on expectation. The theorem includes two constants, $c_1$ and $\lambda$, which are determined by the distribution of non-conformity scores $S$ under the assumptions detailed in Appendix F.3. Similar constants also appear in Theorem 14 of previous work [1]. While these constants are important for the theoretical analysis, our practical method does not require their explicit calculation. Instead, the theorem provides the key insight that guides our approach: optimizing temperature can enhance the efficiency of adaptive conformal prediction. This theoretical foundation is validated by our empirical results in Figure 1.

5. Clarification on computational cost [Q5]:

Thank you for pointing out the potential confusion. For classification, temperature scaling has been an essential part in the workflow of deep learning, as a common practice. Therefore, our method does not require additional computational cost compared to the standard temperature scaling, which also requires optimization [2]. Our method only modifies the training objective of the optimization, rather than introducing a new procedure.

1. Results of non-image prediction tasks [Q1]:

Thank you for this important question. We evaluated ConfTS on text classification using the AG News dataset [1] with a pre-trained BERT model [2] fine-tuned for one epoch. We use APS and RAPS ($\lambda=0.001$) to generate prediction sets, and the results demonstrate consistent improvements, which demonstrates the effectiveness of ConfTS beyond image tasks.

2. Handling distribution shifts with online conformal prediction [Q2]:

Thank you for this insightful question about handling distribution shifts. One promising approach would be to integrate ConfTS with online conformal prediction [3], where the temperature parameter could be dynamically adjusted as new data arrives. This could potentially help maintain optimal prediction set sizes even as the underlying distribution evolves. We believe this represents an interesting direction for future research.

References

[1] Xiang Zhang, et al. Character-level Convolutional Networks for Text Classification. NIPS 2015.

[2] Jacob Devlin, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT 2019.

[3] Isaac Gibbs, et al. Adaptive Conformal Inference Under Distribution Shift. NIPS 2021.

1. The results of ConfTS on ImageNet [W1]:

We clarify that the 'nearly 50% reduction' refers to the average improvement across all six model architectures tested as shown in Table 2. In particular, we conduct experiments on various model architectures: ResNet18, ResNet50, ResNet101, DenseNet121, VGG16, and ViT-B-16. To avoid any misunderstandings, we revised the sentence (lines 25-26) as: 'When averaged across six different architectures, ConfTS reduces the size of APS and RAPS on ImageNet by nearly 50% at an error rate of α = 0.1.'

2. The results on RAPS [W2,3]:

Thank you for raising this important concern about experimental fairness. In our experiment, we set $k_{reg}=1$ because this setting consistently achieves smaller average size in Table 3 of their paper [1]. In addition, we use the same setting to ensure a fair comparison when comparing the average size of RAPS with and without ConfTS. This enables us to analyze the impact of our method in consistent conditions of baselines. As for $\lambda$, we provide extensive results in Appendix I to demonstrate the consistent performance of ConfTS across different $\lambda\in$ {0.002, 0.004, 0.006, 0.01, 0.015, 0.02}.

As you suggested, we also conduct experiments to compare the best performance between RAPS with and without ConfTS with tuned $\lambda$. The results show that ConfTS still achieves improvements for RAPS in average set size:

3. Why we adopt the optimization for searching T [W4]:

In temperature scaling, using optimization to search the temperature value has been a common practice in deep learning [2]. While grid search is indeed a viable approach for scalar optimization, we choose gradient-based optimization for several reasons:

• Grid search can only explore a discrete domain, like $t\in[1.0,0.9,0.8,0.7,0.6]$, which makes it challenging to search for the optimal temperature;
• Grid search can be labor-intensive and requires tuning for each dataset or task;
• Gradient-based optimization can be applied to other non-scalar methods, like vector scaling [2];

Therefore, our method can be treated as an automatic and adaptive tuning method, which requires less manual intervention than Grid search and can be extended to other complex methods of calibration for broad application.

4. Using ConfTr to tune temperature value [W5]:

Thank you for this interesting suggestion. However, ConfTr is not suitable for our setting because it would drive the temperature parameter toward extremely small values. As shown in Figure 1 (c), such small temperature values cause the prediction sets to become overly conservative.

5. The performance of ConfTS on conditional coverage [W6,7,8]:

See general response.

6. Typo in Table 6 [W9]:

Thank you for pointing out this potential source of confusion. To clarify: the rows indicate the methods used to generate the prediction sets, while the columns indicate the scoring functions used in the ConfTS optimization process. We have revised the caption of Table 4.

References

[1] Angelopoulos A, et al. Uncertainty sets for image classifiers using conformal prediction. ICLR 2021

[2] Guo C, et al. On calibration of modern neural networks. ICML 2017.