Reassessing How to Compare and Improve the Calibration of Machine Learning Models

We would like to thank Reviewer Dhyc for taking the time to review our paper, and we appreciate the positive comments regarding our contributions as well the precise suggestions for revisions. We hope to address all of the expressed concerns below.

Weaknesses

The paper explains ECE can be improved by a trivial modification to the confidence function but does not explain why it is a bad thing. If estimating the whole class probabilities is not the objective and we only care about the confidence of the predicted class, this solution could be just fine.

We do not disagree that if we only care about having good (aggregate) confidence calibration and accuracy, there is nothing fundamentally wrong with mean replacement recalibration, i.e. just always using our aggregate confidence. However, the point we wanted to make was that typically people want more than that -- if a model always predicts a constant (or near-constant) confidence, this is likely not useful for downstream decision-makers who would want to make different decisions at different confidence levels.

The descriptions of the method and the results are not clear to me.

In the revision, we have added an explicit discussion in Section 4.2 of how to interpret the components of our plots, which we hope makes the results more clear.

The benefits of the visualization are not clear from the results. In particular, I do not agree with the claim that "looking at the calibration-sharpness diagrams the differences between the two approaches becomes more clear."

We apologize -- we realize now that the example we chose to illustrate the differences between reliability diagrams and our approach is perhaps not the most obvious. We have replaced the comparison in Section 5.2 with a comparison between Histogram Binning and Isotonic Regression, where we think it should be more clear what is to be gained from using the calibration-sharpness diagrams.

The notation in some parts is a little confusing.

We have added some more text to explain notation in parts where it may have been confusing, i.e. Section 4.1.

Some of the proofs lack details, and there are also some writing mistakes. These make the proofs a little difficult to follow.

We have added more detail to all proofs that were pointed out to be too brief, and also fixed typos (thank you for catching them).

Major Comments

Are some of the findings of this paper related to those of Prez-Lebel et al. (2023)?

Thank you for mentioning the work of Perez-Lebel et al. (2023), it is indeed very relevant to our work and we sincerely apologize not discussing it appropriately in the submitted version of the paper. Perez-Lebel et al. (2023) puts forth a similar message to us, but we achieve this using two different approaches (analyzing grouping loss vs. analyzing sharpness). The grouping loss results have to be applied to the confidence calibration problem directly (i.e. reduce to the binary setting where labels are 0, 1 depending on whether we predicted correctly or not), but our results allow us to connect the full generalization error in the multi-class setting with the 1-D confidence calibration error as we discuss in Section 4. We have now referenced Perez-Lebel et al. (2023) in the paper, and have included a more comprehensive discussion in Appendix E comparing our two works.

Corollary 4.7 should be more formally stated.

Corollary 4.7 is now stated fully formally, and the proof also has much more detail than before.

I could not understand the condition $d_{\phi}(x, y) \ge d_{\varphi}(x_i, y_i)$.

Here we recall that $d_{\phi}$ is defined on $\mathbb{R}^k \times \mathbb{R}^k$ whereas $d_{\varphi}$ is defined on $\mathbb{R} \times \mathbb{R}$; i.e. the latter is a "1-D counterpart" to the former. The condition is then that the total divergence $d_{\phi}$ dominates this 1-D divergence applied to any coordinate pair; a simple example is $||x - y||^2$ compared to $(x_i - y_i)^2$ since the former is a sum over the latter for all $i$. We have made this clearer in the discussion preceding Lemma 4.6.

We would like to thank Reviewer sELH for taking the time to review our paper. We are grateful that they found the insights in our paper to be valuable, and we hope to address the pointed out questions and concerns below.

Could you provide specific examples or a more detailed workflow for selecting calibration metrics based on common generalization metrics? A decision flowchart or table mapping generalization metrics to recommended calibration metrics would also be helpful for practitioners applying the proposed framework.

We have revised the text in Section 4 to make it clearer how to practically apply our framework. In particular, we emphasize that all one has to do to obtain a notion of calibration error is to take the generalization error (which is applied to labels $Y$ and model predictions $g(X)$) and then apply that error to the conditional expectation $\mathbb{E}[Y \mid g(X)]$ and the predictions $g(X)$. In this sense there is no need for a table/flowchart, as one can obtain infinitely many different choices of calibration error this way (once we fix our generalization error). Furthermore, most of these calibration errors are not "named" (like ECE) since they are derived directly from the generalization error -- it just happens that the $L^2$ ECE is derived from Brier score.

How could the arbitrariness in selecting kernel types and hyperparameters for calibration-sharpness diagrams be minimized? Could you suggest specific strategies, such as cross-validation approaches or sensitivity analyses, to guide these choices? Additionally, are there recommended default choices for kernels and hyperparameters that perform robustly across various scenarios?

For selecting kernels/kernel hyperparameters, you are absolutely right; the best way to do this would be to have a downstream task/comparison and then tune them with respect to some held-out validation data from that task. We now describe this approach in Appendix H. The tuning of these hyperparameters is trickier when the main product of the method is a visualization, but we can still tune using the estimated calibration error values since those are also produced using the same kernel/kernel bandwidth. As far as recommended defaults are considered, Gaussian and Epanechnikov kernels are standard choices - as we discuss the next point, it was previously investigated in [1] how results actually do not change much as we vary the bandwidth for these choices beyond a certain point.

Given the variability in calibration performance across different kernel types and hyperparameter settings (as shown in the Appendix), could you provide more guidance on appropriate criteria for kernel selection and hyperparameter tuning? This would strengthen the reliability of calibration assessments based on the proposed diagrams.

Although there is variability in the visualizations across different kernels and hyperparameters, we actually interpret the results of Appendix H as showing that for reasonable kernels/hyperparameters the relationships between different model performances are preserved. We understand here that "reasonable" is not well-defined; we now provide more justification in the form of referencing prior work in Appendix H. Historically, people have most frequently compared calibration error using binning estimators with the number of bins being in the range 15-20; while this seems arbitrary, the experiments of [1] show that reported ECE results don't really change as we increase the number of bins beyond this point (but they can change significantly when we decrease the number of bins too much). [1] also provides some explanation for why we can consider taking the bandwidth to be 1/number of bins.

[1] Chidambaram, M., Lee, H., McSwiggen, C., & Rezchikov, S. (2024). How Flawed is ECE? An Analysis via Logit Smoothing. ArXiv, abs/2402.10046.

We would like to thank Reviewer 6vrr for reviewing our paper, and we are glad they found the paper well-presented and the visualization interesting and novel. We hope to address the stated weaknesses below.

1. The experiment section is relatively weak. Only one dataset was examined.

We have now included a Subsection in the Appendix (Subsection F.2) that includes experiments on CIFAR-10 and CIFAR-100; these experiments further corroborate our findings in the main paper.

2. It would be great if the authors could open-source the code and make the new diagrams easier to use for other ML practitioners.

In the original submission of the paper, we included code in the supplementary material that provides all the utilities necessary to recreate our results. After the paper is deanonymized, we will link to an open source version of this code directly in the paper.

We are happy to address any further concerns you may have. We hope that the above, along with the other revisions we describe in the top-level comment, serve to better highlight the usefulness/novelty of our results.

Thanks for the reply. I will maintain my score.