Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing

We thank the reviewer for the time and feedback. We address the questions below.

It would be interesting to see the results of SmoothECE on imbalanced (image) datasets such as SVHN.

We agree that understanding the calibration of real networks in various settings is an important question (and imbalanced settings can be particularly interesting). However, we consider this out-of-scope for our paper, since our paper is about how to define calibration metrics, not about how the metric behaves on ML models of interest (though this is a natural direction for future work).

Theoretically, our SmoothECE theorems do not assume anything about the distribution of predictions, and so will apply for any degree of imbalance. Empirically, the "Solar Flares" experiment (Fig 2b) shows an example of running our method on imbalanced data (the dataset has ~25% positive samples, ~75% negative samples). We can consider adding more imbalenced-data experiments if they would be informative.

The empirical results are missing experiments on more image datasets, especially that cover out-of-distribution scenarios, and different types of networks.

See above; we consider this important but out-of-scope for our paper.

The overall contributions of the paper seem a bit limited as it boils down to applying (Gaussian) kernel to the original distribution for obtaining a smoothed estimate of it (which is already done in the prior work) and a technique to determine the kernel bandwidth.

Note that our contribution is twofold: (1) deriving a theoretically-principled choice of Gaussian bandwidth, which was an unsolved complication in practice, and (2) proving that, with this bandwidth choice, the induced calibration measure satisfies the strong theoretical guarantees of a "consistent calibration measure". We acknowledge that the concept of Gaussian smoothing for reliability diagrams is not new (as we cite on page 3), but the details matter significantly, and it was not previously known that this method could satisfy meaningful theoretical guarantees.

It is a bit difficult to understand that, how the smooth reliability is visually more interpretable than the binned reliability.

We have included a new section (Appendix E, FIgure 3) in the updated pdf, which demonstrates how SmoothECE can behave much better than binned ECE in certain settings.

To what extent, this smoothECE can be trusted as a standalone miscalibration measure?

The goal of our theoretical results is to demonstrate rigorous ways in which smoothECE "can be trusted as a standalone miscalibration measure." Specifically, what it means to be a "good miscalibration measure" was rigorously defined in (Błasiok et al 2023), who called such measures "consistent calibration measures." In this paper, we prove that SmoothECE is in fact a "good" calibration measure in the above rigorous sense – this is our main theoretical result.

We hope this addresses all of your concerns. We are happy to discuss any remaining points further.

We thank the reviewer for their time and feedback, and we are glad the reviewer appreciates the contribution of this work. We address the questions and weaknesses below.

I think it is beneficial to include some synthetic experiments to demonstrate these problems of the Binned ECE estimator and show how the proposed estimator overcomes them.

Thank you for this suggestion. We have now updated the pdf with a new section in Appendix E, including a new Figure (3), to help illustrate the problems with Binned ECE, and the resolution of Smooth ECE. We hope this addresses your concern.

Can the authors comment a bit on the statistical consistency properties of the proposed estimator?

Yes, we have now added an explicit discussion of this in Appendix E.1, which we will reproduce here: Our estimator of SmoothECE also satisfies the classical criteria of statistical consistency. Specifically Theorem 9 shows that as the number of samples $m \to \infty$, the finite-sample estimation error $\varepsilon \to 0$. (In fact, Theorem 9 is stronger than just asymptotic consistency--- it provides a quantitative generalization bound).

On top of page 6, the reflected Gaussian kernel uses sum over is this an infinite sum by the definition of? How is it computed in practice?

Appendix C discusses how the SmoothECE can be easily and efficiently computed. In short, the infinite sum in the definition can be approximated up to an error of $\delta$ by using only $O(\sqrt{\log \delta^{-1}})$ leading terms (in practice: just three such terms are enough). Moreover, even this truncation does not have to be computed explicitly: to approximately convolve function $r :[0,1] \to \mathbb{R}$ with the “reflected gaussian kernel” on [0,1], one can extend it by reflecting around 0 and 1 to residual $\tilde{r} : [-1, 2] \to \mathbb{R}$ (or, more formally $\tilde{r} : [-T, T+1] \to \mathbb{R}$ for some $T\approx \sqrt{\log \delta^{-1}}$), apply FFT to compute convolution with a standard gaussian kernel of such an extended function, and truncate this convolution to interval $[0,1]$. This "reflection+convolution trick" is what we actually use in the code implementation, which is now attached as supplementary material.

Thank you for your detailed review and suggestions. Below, we address each of your points (in the same order as the original bullet points).

Theory/method questions:

• Regarding distinction between $\widetilde{SmECE}$ and $SmECE$: we agree that indeed $SmECE$ is a slightly more natural notion as a calibration measure. Those two notions are close to each other, both in spirit, as well as formally (see for instance Lemma 10 in appendix A). On the other hand, the $\widetilde{SmECE}$ is more directly reflected in a visually interpretable reliability diagram, which we argue for constructing. As such, we chose to define both those notions and highlight the close relations between those two, while arguing for SmECE to be the more fundamental construction, which should be eventually used to report calibration error itself. A slightly more detailed discussion about this is present in Appendix A.
• Choosing the fixed point $\sigma^*$ is to a small extent arbitrary. Lemmas 7 and 8 provide an upper and lower bounds of the distance to the nearest perfectly calibrated predictor in terms of $smECE_\sigma$ and $\sigma$. This leads to a range of parameters $\sigma$ for which those upper and lower bounds are good enough for the measure to satisfy the “consistency” requirement. One could, for instance, choose $\mu(D) := \min_{\sigma} (smECE_{\sigma}(D) + \sigma)$. Such a $\mu$ would also be a consistent calibration measure. Yet it wouldn’t necessarily provide a specific scale $\sigma$ for which $smECE_{\sigma}(D)$ is proportional to $\mu(D)$ — and the quantity related with $smECE_{sigma}$ is the one we are presenting on the reliability diagram. On the other hand, choosing a fixed point $\sigma_*$, provides an approximate minimizer for $smECE_{\sigma}(D) + \sigma$ (up to a factor of two, since $smECE_{\sigma}(D)$ is non-increasing with sigma), provides a scale on which $smECE_{\sigma}(D)$ (which is visualized on the diagram) is proportional to $smECE_{\sigma}(D) + \sigma$ (which is guaranteed to upper bound the distance to calibration), is easy to compute, and is in a sense “canonical” in a range of parameters sigma for which we would have the consistency property — a choice that can be universally agreed on, to end up with a parameter-free calibration measure with respect to which different classifiers can be easily compared.
• Thank you for pointing out that the importance of consistent calibration measures was not discussed enough, that is a great point. We have included additional discussion in the current write-up (Appendix E), which we will consider moving into the main body in the full version (where we will have more space).
• The more precise upper and lower bounds between smECE and distance to the nearest perfectly calibrated predictor are given at the end of Section 3.3 (giving $\alpha_1 = 1, \alpha_2=1/2$). This provides a decent continuity guarantee for the smECE measure. We provided some synthetic examples in Figure 3, where this continuity is highlighted — binning ECE and the associated binning diagram behaves extremely differently on two very close datasets of (prediction, outcome), while smECE provides a stable picture of mis-calibration.
• Regarding intuition behind Lemma 7 and Lemma 8. Lemma 7 is best understood in terms of duality theorem for distance to calibration: such a distance is (up to a constant factor) equal to maximum over all Lipschitz functions $w: [0,1] \to [-1,1]$ of a correlation between $w$ applied to the prediction $f$, and the residual between prediction $f$ and outcome $y$, i.e. $\sup_{w} \mathbb{E}[w(f) (f-y)]$ (in this paper this latter quantity is called weak calibration error, see Definition 19 and Theorem 20 in Appendix D.1). With this in hand, it’s not that surprising why quantities like $(\widetilde{smECE}_\sigma + \sigma)$ upper bound the weak calibration error. For any given Lipschitz function $w$, we can bound the $\mathbb{E}[w(f)(f-y)]$ in terms of $ECE(\pi_R(f+Z), y)$ and $\sigma$ where $Z$ is a an independent gaussian with standard deviation sigma. Indeed, while extending the notion of smECE to non-trivial metrics on the space of predictions $[0,1]$, one of the main steps we had to take care of was generalization of the duality theorem (see Appendix B.1 and D.8). Lemma 8 follows from observing that smECE(D) is Lipshitz-continuous with respect to the Wasserstein metric on distribution D (see Lemma 23 for a formal statement of this). Then, by comparing smECE(D) to smECE(D'), where D' is the closest perfectly-calibrated distribution to D, we obtain the bound in Lemma 8.

We thank the reviewer for their time and feedback. We have implemented several of your suggestions, and we respond to your questions below.

• Re. discontinuity problem: Thank you for your suggestion – it will surely improve our paper to provide experimental data highlighting the issues with discontinuity of BinnedECE and associated reliability diagrams, as well as showing how those problems disappear when using our method instead. We have included such a diagram now, in Appendix E (Figure 3) of the updated pdf.

• Re. code: We have now uploaded the anonymous code here: https://anonymous.4open.science/r/smoothece-anon/ (also in the attached .zip file), as we mention in our common response.

• Thank you for pointing out the inconsistency in Figure 2, plot (d). The shaded areas (and the thickness of the red line) indeed were meant to indicate the kernel density estimates. The plot (d) was produced by using our library with a different option enabled, showing the confidence intervals for the red line. We will change this back to the standard format, so that it is consistent with other plots. (We chose not to focus on confidence intervals in this paper in order to provide cleaner plots, but they are an option supported by our library).

• Re. differentiability: Yes, on any fixed scale $\sigma$, the $SmoothECE_{\sigma}$ is differentiable. Thus it is possible to include it in the loss while training classifiers. Our current library is written in numpy, but it should be easy to port to pyTorch/JAX, and perform the differentiation using the standard autodiff. This is an interesting idea for future work.

We hope this addresses the reviewer's concerns, and we are happy to discuss any remaining issues further.