A Benchmark Study on Calibration

Reference

[1] Sunil Thulasidasan, et al. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.

[2]Mukhoti, Jishnu, et al. "Calibrating deep neural networks using focal loss." Advances in Neural Information Processing Systems 33 (2020): 15288-15299.

[3] Wang, Deng-Bao, Lei Feng, and Min-Ling Zhang. "Rethinking calibration of deep neural networks: Do not be afraid of overconfidence." Advances in Neural Information Processing Systems 34 (2021): 11809-11820.

[4] Wang, Deng-Bao, et al. "On the Pitfall of Mixup for Uncertainty Calibration." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[5] Calibration Bottleneck: What Makes Neural Networks less Calibratable? “ICLR 2024 Conference Submission3477”

[6]Thulasidasan, Sunil, et al. "On mixup training: Improved calibration and predictive uncertainty for deep neural networks." Advances in Neural Information Processing Systems 32 (2019).

[7] Hendrycks, Dan, et al. "Augmix: A simple data processing method to improve robustness and uncertainty." arXiv preprint arXiv:1912.02781 (2019).

[8] Liu, B., Ben Ayed, I., Galdran, A. and Dolz, J., 2022. The devil is in the margin: Margin-based label smoothing for network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 80-88).

[9] Patra, R., Hebbalaguppe, R., Dash, T., Shroff, G. and Vig, L., 2023. Calibrating deep neural networks using explicit regularisation and dynamic data pruning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1541-1549)

[10] Hebbalaguppe, R., Prakash, J., Madan, N. and Arora, C., 2022. A stitch in time saves nine: A train-time regularizing loss for improved neural network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16081-16090).

[11] Kumar, Aviral, Sunita Sarawagi, and Ujjwal Jain. "Trainable calibration measures for neural networks from kernel mean embeddings." International Conference on Machine Learning. PMLR, 2018.

[12] Bohdal O, Yang Y, Hospedales T. Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Error[J]. arXiv preprint arXiv:2106.09613, 2021.

Re significance of Section 4.4: In Section 4.4, our observation highlights that a well-calibrated model with superior pre-calibration ECE may not necessarily achieve better post-hoc calibration performance than a poorly calibrated model with a higher pre-calibration ECE. For instance, in the table below, Model A exhibits a better pre-calibration ECE than Model B but offers less room for improvement through post-hoc methods, resulting in a poorer post-hoc ECE than Model B. Given that post-hoc calibration methods, such as temperature scaling, are both computationally efficient and effective, prioritizing an overall better calibration performance, or "calibratability," deserves more attention. Our observation supports the "calibratable" objective proposed in [1][2][3], suggesting that research efforts should prioritize achieving models with superior post-hoc calibration performance.

Re ‘data pre-processing’ methods: By “data pre-processing”, we mean data augmentation methods such as mixup[6] and Augmix[7], we apologize for the confusion using “data pre-processing”.

Re help to develop to new calibration method:

• This paper can benefit the calibration research in different ways, for example: As an illustration, delving deeper into the architecture design reveals insights into achieving better calibration. For instance, as depicted in Figure 10, a discernible trend emerges, indicating that better-calibrated models exhibit a preference for Conv33 in edge 1 over Conv11 and favor a residual connection in edge 4 within the NATS-Bench scope. This implies that a better-calibrated model may lean towards incorporating larger kernel sizes in the early layers of a CNN block.
• One can assess the validity of newly proposed metrics by evaluating their consistency with the set of metrics we have examined. Specifically, using the provided checkpoints and the 11 different metrics we assessed, a newly proposed metric can gauge its consistency by evaluating the checkpoints and calculating the Kendall ranking coefficient in comparison to the existing metrics. If the results exhibit little correlation or are negatively correlated with the established metrics, it suggests that the new metric warrants more careful analysis and scrutiny.

Re Analysis on reliability of calibration metrics: We agree that current analyses are hard to measure the reliability of calibration metrics. However, this work reminds researchers to avoid using certain ambiguous metrics such as equal mass class-wise ECE and AuC on OoD datasets.

Re bin-size impact on train-time calibration: We conduct a toy experiments on bin-size involved train time calibration algorithm DECE[12] and observe that different bin size can bring larger impact on the post-hoc ECE, where small bin size works better for DECE, as shown in the table

We appreciate the valuable comments from Reviewer qRJE

Re Expansion of existing studies: We agree studies are the expansion of current research, however, they provide some different results from previous study. For example,

• Previous works use AuC on OoD datasets as a measurement of calibration[1][2]. However, in Section 4.2 we found that there is no correlation between the AuC and ECE, which indicate AuC on OoD may not be a reliable calibration measurement.
• In section 4.4, we observe that a well calibrated model may not achieve better post-hoc calibration performance than a poor calibrated model. This observation supports the “calibratable” objective proposed in [1][2][3] in a way that research should focus more on obtaining a model with better post hoc calibration performance. Since the post-hoc calibration methods like temperature scaling are computationally cheap and effective, an overall calibration performance worth more attention. See more details in section "Re significance of Section 4.4".
• Section 4.6 study the relationship between accuracy and calibration and section 4.7 study the architecture design for better calibration performance, which are neither studied before from our understanding.

Re other calibration methods: Following the suggestion, we incorporated Focal Loss[2] and MMCE Loss[11] as additional calibration methods. We trained six human-designed CNN models—ResNet18, ResNet34, ResNet50, ResNet110, Wide-ResNet-26-10, and Densenet121—on CIFAR-10 and CIFAR-100 using Focal Loss and MMCE Loss, both recognized as classic train-time calibration methods. Our observations, consistent with those reported in Appendix B, are detailed in Figure 16. As depicted in Figure 16 and similar to Figure 2, models trained using different train-time calibration methods all exhibit little correlation between results on CIFAR-10 and CIFAR-100. This suggests that the calibration properties of a specific architecture may not generalize effectively across different datasets. In Figure 17, ECE evaluated across different bin sizes shows little correlation between pre and post temperature scaling, indicating that well-calibrated models do not necessarily show improved calibration performance after post-hoc calibration techniques. This trend is particularly pronounced on CIFAR-100, where post-hoc calibration performance becomes negatively correlated with pre-calibration performance. Additionally, we observe that bin size can significantly impact post-hoc calibration performance, aligning with the observations in Section 4.4. In terms of the reliability of calibration metrics, we conducted an analysis of the correlation between all calibration metrics, as presented in Figure 14 in Appendix A and Figure 18 and Figure 19 in Appendix B. Notably, equal-mass classwise ECE exhibits a different pattern compared to other metrics, especially on CIFAR-100, reinforcing the observations outlined in Section 4.3.

Re consistency between metrics and preferred calibration metrics: Although some other metrics are theoretically different from ECE, they share the same objective which aims to measure the alignment between the true likelihood and the predicted confidence. Thus, ECE shows consistent results with other metrics. However, this consistency is significantly influenced by the way how to approximate this alignment, such as the binning scheme. We mainly attribute the gap between class wise ECE and other metrics to the scenarios where the equal mass binning strategy aims to distribute samples uniformly across bins in a multi-class setting. This approach often leads to a scenario where the negative class is predominant. In practical applications of multiclass prediction, it's common for the model to assign very low confidence scores to a majority of the classes, often more than 95 out of 100. These low scores, hovering close to zero, indicate an absence of uncertainty. Equal mass class wise ECE will assign almost all bins to these near-zero confidence samples, except for one. Thus, equal mass binning experiences a significant decrease in estimation accuracy, making it unreliable. Thus, although all these metrics share the same objective, some of them cannot provide a precise estimation of this alignment in some scenarios. Considering this fact, we conduct the evaluation of the reliability of calibration metrics as in section 4.3, which we observe the unreliability of equal mass class wise ECE. On the other hand, the consistency between different calibration metrics with different theories such as ECE and MMCE support the reliability of metrics against each other. In the current stage, we recommend the use of calibration metrics that are more consistent with each other, such as ECE and MMCE.

I thanks authors for responding to my comments. The responses to a majority of comments are satisfactory, including evaluation with train-time calibration methods, bin size impact, and significance of sec. 4.4. However, the following points require better explanation: 1) how the proposed study can encourage new research in this regard, and on the 2) expansion of existing methods. For instance, 1) is mostly missing description about the potential of developing new calibration methods with this study.

Respected reviewer, should you have any further concerns, I am eagerly anticipating your response.

Respected reviewer, should you have any further concerns, I am eagerly anticipating your response.

We appreciate the valuable comments from Reviewer Xzoe

Re NAS + Calibration: To avoid potential misunderstanding regarding the term "NAS + Calibration", please let us clarify that this study does not involve conducting neural architecture search (NAS) specifically for calibration purposes. The primary focus of this research is the examination of calibration properties. To address calibration-related research questions, such as the reliability of calibration metrics, one approach is to assess the consistency of different metrics based on a substantial number of well-trained models. However, collecting such a substantial dataset is often challenging due to the associated training costs. Fortunately, NATS-Bench [1] provides access to 117.9K well-trained models with various architectural designs, enabling us to conduct a comprehensive and generalisable study.

Re datasets: Modern large-scale datasets, such as LAION, primarily serve image-text alignment tasks, notably in training stable diffusion models. However, it's worth noting that calibration tasks typically revolve around classification and regression. In recent years, standard benchmarks for calibration studies have included well-known datasets like CIFAR-10, CIFAR-100, and ImageNet [2] [3] [4] [5]. While the prospect of applying calibration tasks to LAION is intriguing, it falls outside the scope of the present work.

[1] Dong, Xuanyi, et al. "Nats-bench: Benchmarking nas algorithms for architecture topology and size." IEEE transactions on pattern analysis and machine intelligence 44.7 (2021): 3634-3646.

[2] Tao, Linwei, Minjing Dong, and Chang Xu. "Dual Focal Loss for Calibration." arXiv preprint arXiv:2305.13665 (2023).

[3] Ghosh, Arindam, Thomas Schaaf, and Matthew Gormley. "AdaFocal: Calibration-aware Adaptive Focal Loss." Advances in Neural Information Processing Systems 35 (2022): 1583-1595.

[4] Minderer, Matthias, et al. "Revisiting the calibration of modern neural networks." Advances in Neural Information Processing Systems 34 (2021): 15682-15694.

[5] Gawlikowski, Jakob, et al. "A survey of uncertainty in deep neural networks." Artificial Intelligence Review 56.Suppl 1 (2023): 1513-1589.

Datasets Section: While previous papers use these tiny datasets, I as a reviewer, object to use them in modern research. The reason is that statistically speaking, they are small, and as many other systems nowadays have shown (e.g., CLIP, OpenCLIP, etc..), large and high-quality datasets are important. To me, the use of these datasets do not represent the real scenarios of machine learning; mainly because their image resolution is quite small, their scale is tiny, and the scenarios they cover are niche. If we don't aim to show results on large-scale datasets, I don't expect the field in making leaps forward in understanding machine learning.

Dear Reviewer Xzoe,

Thank you for taking the time to review our paper. We appreciate your valuable feedback and suggestions. Below are our responses to the points you raised:

1. Calibration Focus:

Original Comment: "The primary focus of this research is the examination of calibration properties. This is not clear in the narrative. Please fix."

Revised Response: We have address this concern by refining our statement to explicitly state that the primary objective of our research is to investigate calibration properties, as modified in the fourth paragraph of introduction.

2. Datasets Section:

In our examination of 249 recent calibration papers, we discovered that none of them—0 out of 249—involved training or evaluating models on LAION. The fact that most recent calibration papers are not using LAION, which indicates that LAION is not a widely used dataset in the calibration community, and using widely adopted CIFAR10, CIFAR100 and ImageNet can prove the effectiveness of calibration algorithms as other accepted calibration papers[2][3][4][5].

Specifically, to delve into the use of large-scale datasets, specifically LAION, in calibration research, we scrutinized 249 calibration-related papers across ICLR2023 submissions, ICLR2024 submissions, NIPS2023 accepted papers, and ICML2023 accepted papers. The criterion for calibration-related papers was the inclusion of the keywords "calibration," "confidence," or "uncertainty" in the title. Our investigation revealed that one[1] of these papers cited LAION, and none utilized LAION for experiments. The web crawler code can be found in this anonymous code link. The detailed statistics are shown in the table below.

If you have further concern, please don't hesitate to comment.

Thank you once again for your insightful review.

Best regards,

Submission 1053 Authors

References

[1] CONFIDENCE-AWARE REWARD OPTIMIZATION FOR FINE-TUNING TEXT-TO-IMAGE MODELS (ICLR2024 submission)

[2] Tao, Linwei, Minjing Dong, and Chang Xu. "Dual Focal Loss for Calibration." arXiv preprint arXiv:2305.13665 (2023).

[3] Ghosh, Arindam, Thomas Schaaf, and Matthew Gormley. "AdaFocal: Calibration-aware Adaptive Focal Loss." Advances in Neural Information Processing Systems 35 (2022): 1583-1595.

[4] Minderer, Matthias, et al. "Revisiting the calibration of modern neural networks." Advances in Neural Information Processing Systems 34 (2021): 15682-15694.

[5] Gawlikowski, Jakob, et al. "A survey of uncertainty in deep neural networks." Artificial Intelligence Review 56.Suppl 1 (2023): 1513-1589.

Respected reviewer, should you have any further concerns, I am eagerly anticipating your response.

We appreciate the valuable comments from Reviewer S1Jc

Re findings generality: While our analysis in the main text primarily focuses on CNN models, we extend our investigation to transformer-based models, as detailed in Appendix A. In further support of our study, we trained six human-designed CNN models—ResNet18, ResNet34, ResNet50, ResNet110, Wide-ResNet-26-10, and Densenet121—on CIFAR-10 and CIFAR-100, obtaining similar conclusions as reported in Appendix B. As depicted in Figure 12 in Appendix A and Figure 17 in Appendix B, the ECE across different bin sizes demonstrates little correlation between pre and post temperature scaling for both Transformers and CNNs. This suggests that well-calibrated models do not necessarily exhibit improved calibration performance after post-hoc calibration techniques. This phenomenon is particularly pronounced on CIFAR-100, where post-hoc calibration performance becomes negatively correlated with pre-calibration performance. Regarding the impact of bin size, we observe a substantial influence on post-hoc calibration performance, aligning with the observations outlined in Section 4.4. In terms of the reliability of calibration metrics, we conducted an analysis of the correlation between all calibration metrics, as presented in Figure 14 in Appendix A and Figure 18 and Figure 19 in Appendix B. Notably, equal-mass classwise ECE exhibits a different pattern compared to other metrics, especially on CIFAR-100, reinforcing the observations outlined in Section 4.3.

Re NLP Task: Recognizing the significance of task diversification, we undertook an NLP classification experiment using the 20 newsgroups dataset. Our experimental setup included 2 CNN-based models, 1 MLP-based model, 2 RNN models, and 1 BERT model. Each model is trained for 20 epochs. Notably, the correlation among the six models between pre-calibration ECE and post-hoc ECE was found to be -0.13. This aligns with the observation in Figure 4b, indicating that the impact of post-hoc calibration methods is not uniform across all models.

Re NAS search space: In an effort to extend the applicability of our observations beyond a limited search space, we conducted experiments on human-designed Transformers and CNNs. The outcomes of these experiments align with the findings in the main text, as detailed in Appendices A and B.

Re hyper-parameters fine tune: While we acknowledge that hyperparameter tuning for each model could potentially yield more precise results, the vast number of models involved in our study (117,702 unique neural networks) makes this approach computationally intensive. As a pragmatic tradeoff, we opted to establish a fixed and fair set of hyperparameters for all models to conduct our experiments.

Re border impact: Our several observations can help improve the research in calibration and accuracy, for example,

• In section 4.1, we suggest testing model calibration performance on downstream tasks specifically, since the fact that a well calibrated model on source tasks does not necessarily perform well on downstream tasks.
• In section 4.2 and 4.3, we suggest avoiding equal mass class wise ECE or AuC on OoD datasets for calibration evaluation, which helps the calibration evaluation.
• In section 4.4, we observe that a well calibrated model may not achieve better post-hoc calibration performance than a poor calibrated model. This observation supports the “calibratable” objective proposed in [1][2][3] in a way that research should focus more on obtaining a model with better post hoc calibration performance.
• In section 4.5, we observe the tradeoff between accuracy and ECE, which can facilitate the research in prediction accuracy.
• In section 4.6, the impact of bin-size might support the study of those train-time calibration methods which involve the hyperparameter bin size, such as SoftECE[4] and DECE[5].
• In section 4.7, the analysis on architecture design of better calibration models can support the research on calibration from the view of architecture design.

[1] Wang, Deng-Bao, Lei Feng, and Min-Ling Zhang. "Rethinking calibration of deep neural networks: Do not be afraid of overconfidence." Advances in Neural Information Processing Systems 34 (2021): 11809-11820.

[2] Wang, Deng-Bao, et al. "On the Pitfall of Mixup for Uncertainty Calibration." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[3] Calibration Bottleneck: What Makes Neural Networks less Calibratable? “ICLR 2024 Conference Submission3477”

[4] Karandikar A, Cain N, Tran D, et al. Soft calibration objectives for neural networks[J]. Advances in Neural Information Processing Systems, 2021, 34: 29768-29779.

[5] Bohdal O, Yang Y, Hospedales T. Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Error[J]. arXiv preprint arXiv:2106.09613, 2021.

Respected reviewer, should you have any further concerns, I am eagerly anticipating your response.