When is Multicalibration Post-Processing Necessary?

We address the weaknesses and questions below.

W1: Regarding observation 3. The association between the accuracy and the calibration fraction is somewhat clear for HMDA but it’s definitely unclear of the other datasets (Appendix F.3). Accuracy seems to even increase as the calibration fraction increases on certain datasets. Thus, the authors conclusion in observation 3 is only based on HMDA. I suggest dropping it or providing more support for that. The current experimental results do not support it.

Note that in Observation 3, we say “a practitioner utilizing multicalibration post-processing may have to trade off worst-group calibration error and total accuracy.” We do not claim that this tradeoff exists in all cases. This is an important note to practitioners, in particular because they may not have been previously aware that they may need to make such a tradeoff. For example, we would not want to make the claim that “a practitioner utilizing multicalibration post-processing may NOT have to trade off worst-group calibration error and total accuracy.”, even though such a claim is supported in some cases by the data.

The tradeoff we observe here tends to occur for larger calibration fractions. We do often observe accuracy improve as calibration fraction moves from 0 to 0.4, but that accuracy then deteriorates as calibration fraction goes from .4 to 1.0. See the following examples in Appendix F.3: MLP on Credit Default (p. 33), MLP on MEPS (p. 33), MLP on HMDA (p. 33), Decision Tree on Credit Default (p. 34), Decision Tree on Bank Marketing (p. 34), Random Forest on Bank Marketing (p. 35), SVM on MEPS (p. 37), Naive Bayes on Credit Default (p. 38). We thank the reviewer for raising this point, however, and we will make all this discussion more explicit in the next version.

W2: Regarding observation 3. Max smECE of ERM is about twice as large as the max smECE after calibration in case of Naive Bayes, SVM, and decision trees. Then how do the authors justify their observation?

We assume that the reviewer is referring to “after [multi]calibration” for NB, SVM, and DTs. Indeed, we agree with the reviewer that multicalibration can be helpful; we mention this in Observation 2: “HKRR or HJZ post-processing can help un-calibrated models like SVMs or Naive Bayes”, where we also provide empirical justification. In Observation 1, we point out that models which are multicalibrated out of the box also tend to be multicalibrated (on tabular data), but the converse also has merit: mis-calibrated models are also likely mis-multicalibrated. We observe that Naive Bayes, SVMs, and decision trees tend to be miscalibrated out of the box (see Fig. 2, Appendix F.2), and that multicalibration post-processing can help them. We will improve clarity and expand this discussion in the next version.

W3: Regarding observation 3. Along the same lines, observation 7 suggests that there is a substantial difference between the results for language/vision data and tabular data because in the latter we do not see such gains. As mentioned right above, Figure 2 shows such gains. It just depends on which algorithms we look at.

We apologize for the confusion, and will work to improve clarity. As you observe, the NNs trained on vision/language data really do benefit from multicalibration post-processing. Some models trained on tabular data also do benefit from multicalibration post-processsing. Note, however, that the MLPs trained on tabular data do not tend to benefit from multicalibration post-processing in a statistically meaningful way. In any case where we draw a direct comparison between vision/language models and tabular models, we meant to compare only the NNs trained on these different types of data.

More broadly, we mean to convey that multicalibration outcomes are both model and data dependent. We will improve the clarity in the next draft and remove this confusion for future readers. Thank you for pointing this out!

W4: Other comments. Observation 5 about the robustness of HJZ to hyperparameter choices does not hold for language and vision data? How do the authors explain this finding?

We kindly refer to our global response for this discussion.

W5: Minor details. What are the error bars in Figure 3? Standard deviations? Do the \pm values in Figure 3 represent standard error or deviation?

The error bars and $\pm$ values throughout our paper always represent standard deviation. We will make this explicit in the next version. We thank the reviewer for the helpful comments and feedback to our work. We will clarify the mentioned discussion in the next version.

We thank the reviewer for their time and thoughtful feedback. If the pressing concerns have been sufficiently addressed, we kindly ask the reviewer to consider adjusting their score by taking the rebuttal into account. Thank you!

            for m in metric_suffixes:
                # mean
                s = sum([run[m] for run in metrics[cf][alg_type][param_idx]])
                n = len(metrics[cf][alg_type][param_idx])
                mean[m] = s / n

                # standard deviation
                sq_diff = sum([(run[m] - mean[m])**2 for run in metrics[cf][alg_type][param_idx]])
                sd[m] = (sq_diff / n) ** 0.5

We agree that recent theoretical consequences of multicalibration have provided more than sufficient motivation for an empirical analysis. In light of this observation, we point out that another contribution of our work is the benchmarking repository containing all experimental code (submitted with the supplemental material). We believe this will lower several barriers to future work in this direction.

We address the weaknesses and questions below.

We agree that a more thorough treatment of [1] is needed; this will be included in the next version. The connection to our work is subtle. In [1], the authors consider performing ERM over some C’ (superset of C), so that the resulting model is multicalibrated with respect to C. In their case, C’ is a family of neural networks (NNs), and C is some family of smaller NNs. In many of our experiments, we indeed perform (approximate) ERM over some family of NNs, and one might expect this to result in multicalibration with respect to our finite collection of groups G, which are easily computable by some class of smaller NNs. We find this to largely be the case in our tabular experiments (Observation 1). This is, as you said, one unique aspect of neural networks vs other model families: for NNs, it is therefore possible to obtain multicalibration guarantees over arbitrarily-complex groups by simply taking "large enough" networks (without making design choices specific to the group structure). We will clarify these points in our revision.

We also agree that a more faithful account of connections to robustness is in order, especially in light of recent theoretical work drawing some connections between these areas [2, 3]. We note that [2] considers corruptions in subgroups of the training data rather than general distribution shift. In general, we view multi-group robustness as a notion of robustness which, like multicalibration, aims to “respect” multiple groups simultaneously. Perhaps the closest connection between multigroup robustness and multicalibration is that they intuitively may share similar mechanisms (as the reviewer noted). That is, the mechanistic question in both cases is to understand which groups can be easily computed from the predictor's "features" – ie, which groups are easy for the predictor to distinguish. We will elaborate on this point (and the relations to the mentioned works) in our revision.

Apart from this similarity, we think it is necessary to delineate practical multicalibration concerns from practical robustness concerns. As noted in Appendix C.2, it is common for empirical works in subgroup robustness to consider groups of a fixed label, and to consider worst group accuracy. These groups alone are not meaningful from the perspective of multicalibration, however, as multicalibration requires more refined probability estimates even when expected labels lie in (0,1). We will explain this relationship and cite [4] (with further discussion) in the next version of our paper.

[1]: Błasiok et al. Loss minimization yields multicalibration for large neural networks. [2]: Hu et al. Multigroup Robustness [3]: Wu et al. Bridging multicalibration and out-of-distribution generalization beyond covariate shift. [4]: Rosenfeld et al. (Almost) provable distribution shift via Disagreement Discrepancy.

W2: Findings may not hold for "non-standard" group definitions.

We give a detailed discussion of this in our global reviewer response, to which we kindly refer.

Briefly, the fact that our results don't hold for non-standard groups is inherent to the problem formulation. It is impossible for a fixed predictor to be multicalibrated with respect to all groups (as you point out), and thus we must decide which groups to consider when studying multicalibration. Certain theory works chose to consider "all groups computable by a small circuit/DNN" (e.g. [Blasiok et al, 2023] and [Hébert-Johnson et al. 2017]). Since our focus is on the application side, we instead chose groups that are relevant to fairness applications, which are often "intuitive" groups as you said.

At a high level, the prior theory works and our empirical work use group definitions at two ends of the spectrum (from "all circuits" to "intuitive groups"). We could hope to eventually understand exactly which groups our algorithms are multicalibrated on, but this goal does not yet appear in reach; our work is just the first step.

We will be sure to clarify these limitations, and the importance of group definitions, in our revision.

W3: Discrepancy in HKRR / HJZ guarantees and calibration measurement.

We agree that the distinction between the theoretical guarantees of HKRR / HJZ (via ECE / L-p calibration) and what we measure (smECE) is important. We report both ECE and smECE for this reason, but will do better to specify which of our takeaways / observations are particular to the metric of interest.

W4: Other issues: "Besides not enunciating group choices in the main part as just discussed, the algorithms that are actually used are not explained in any sort of pseudocode, nor are their objectives/claimed bounds ever stated"

We intend to add two more discussions to our next version: (1) a clear high-level description of the two families of multicalibration algorithms and their guarantees (including what kind of calibration metric they guarantee performance on), as well as psuedo-code for the algorithms (most likely relegated to the appendix); and (2) a description of how and why we selected the groups for each dataset. We will also include a more explicit discussion about the limitations of our work, regarding the questions you posed in W2.

We thank the reviewer for their time and thoughtful feedback. If the pressing concerns have been sufficiently addressed, we kindly ask the reviewer to consider adjusting their score by taking the rebuttal into account. Thank you!

We thank the reviewer for the detailed comments and feedback. We will address the weaknesses and questions.

W1: The paper could be more self-contained. While it's understandable for some results to be in the appendix given the number of experiments run in this paper, I worry that too much may have been pushed to the appendix (though this may be unavoidable).

We agree with this point, and will include more figures in the body of our next version, and experiment with compressing some of the data presented in the main paper. Under each figure, we will also include a direct reference to related appendices. We believe that we can further compress some of the experimental results and (especially) figures in order to include more in the main paper.

W2: Some details, such as how groups are chosen for language tasks, are also not clear when consulting the appendix, particularly for readers who are not familiar with how these groups are normally chosen. Details like this would help readers to better understand the contribution of the paper.

We also agree with this point. We will add a clear description of our group selection criteria to the body, as well as a high-level description (and pseudocode in the appendix) of the multicalibration algorithms considered.

We determined groups by “sensitive” attributes—individual characteristics against which practitioners would not want to discriminate. In many cases, such attributes naturally included race, gender, and age, and then varied with available information. This included all tabular datasets as well as the CelebA dataset (image). On the other datasets, samples are not necessarily in correspondence with individuals, (e.g. postings on an internet platform, or slides of tissue).

On datasets where samples are not in correspondence with individuals—Camelyon17, Amazon Polarity, and Civil Comments—we define groups based on available information that can be viewed as “sensitive” with respect to the underlying task. In other words, we define groups such that an individual or institution using a predictor which is miscalibrated on this group may be seen as discriminating against the group. Ideally, a social media service will not be underconfident when predicting the toxicity of posts mentioning a minority identity group; such predictions may allow hate speech to remain on the platform. Similarly, we would want a shopping platform to promote product listings fairly: a predictor which outputs mis-multicalibrated product ratings may boost listings for technology products proportionally to their true ratings but unfairly ignore the positive reviews of book listings.

Aside from these heuristics, we also required that groups composed at least a 0.005-fraction (.5%) of the underlying dataset. This imposed a degree of uniformity in our group selection across all datasets.

Q1: What are the groups for text experiments?

The text datasets are Amazon Polarity and Civil Comments. Groups for these datasets can be seen in Appendix C.6 (page 19) but briefly can be described as the presence of certain relevant words or phrases. We will also improve the group explanations in the main text to avoid confusion.

Q2: Figure 2 claims that MLPs are among the models that do not benefit from post-processing, however the table appears to show better multicalibration after post-processing. Is this claim correct?

In the table of Figure 2, in the first group of 4 rows, it can be seen that MLP HJZ obtains a max smECE of 0.076 ± 0.018, while MLP ERM obtains a max smECE of 0.086 ± 0.015. While we agree that the max smECE of HJZ is less than that of ERM here, this difference is not statistically meaningful: each of these means is within its standard deviation of the other.

We give a more detailed discussion of this point in response to Q4 of reviewer Zyam, to which we kindly refer.

We thank the reviewer for the detailed comments and feedback. Please kindly refer to the global response for Q2 and Q5. We address the other weaknesses and questions below.

W1: We agree with the reviewer that we do not introduce any new algorithms. The expressed purpose of our work is to empirically study existing algorithms and attempt to determine their potential for practical use in a host of important (existing) regimes. There are already a number of theoretical works examining and proposing multicalibration (MCB) algorithms which we cite and discuss (Hébert-Johnson et al., Gopalan et al. 2022a/b, Bastani et al. 2022 etc.). We believe that determining when and where these algorithms are useful in practice is an extremely important next step for the research area.

More broadly, we respectfully disagree that focusing on existing algorithms is a weakness of papers in machine learning. It is worth noting the existence of several important empirical studies whose main contributions were not novel methods or theoretical results. For example, the main contributions of the important works [1, 2, 3] are almost purely focused on examining existing algorithms from an empirical lens. Such evaluation research is especially important for algorithms involved in high-stakes decision making in the real world (including multicalibration e.g., Barda et al. 2021).

[1]: Grinsztajn et al., Why do tree-based models still outperform deep learning on typical tabular data? [2]: Zhang et al., Understanding Deep Learning Requires Rethinking Generalization. [3]: Minderer et al., Revisiting the Calibration of Modern Neural Networks.

W2: We agree that we do not provide much explanation for the reasons behind the observed phenomena. However, we envision our work as an important step to understand when and where to focus additional study in order to better understand the applicability of MCB algorithms. Before our work, it was not at all obvious when and where MCB may or may not be helpful to the standard machine learning practitioner. Importantly, we believe that we already provide numerous helpful insights and words of caution.

W3: We agree that many experiments are relegated to the appendix. This was due to space constraints: our goal was to have a cohesive message from over 40K MCB runs in the main body of the paper. We hoped to distill practical take-aways and best-practices for practitioners wishing to apply and measure the effectiveness of MCB post-processing algorithms. If accepted, we plan to better compress experimental results from the appendix and include them in the main body, with in-line references to appendix links in the captions.

Q1: HKRR / HJZ are efficient in that the number of samples required is polynomial in the parameters of the algorithm. However, the degree of this polynomial may be large. For example, sample complexity bounds of the HKRR algorithm are $O(1/(\alpha^4 \cdot \lambda^{1.5}))$. Unfortunately, this quickly grows to the order of millions of samples for moderately small choices of $\alpha$ (multicalibration error) and $\lambda$ (bucket width). We view our work as investigating whether we really need this large number of samples to empirically provide reasonable multicalibration levels in practice.

Q3: We vary the amount of (hold-out) calibration data widely due to the sample complexity issues discussed in Q1. In particular, the required samples to guarantee theoretical convergence is very large, and a priori, it was not clear at all how much data should be given to MCB algorithms in practice. This is because MCB algorithms generally require many more samples than simple calibration (which often performs well with only 10% of the data). Indeed, we find that in some cases for tabular data, MCB performs best with 40-80% sized calibration sets. More generally, this is due to a trade-off between the effectiveness of MCB post-processing and the (latent) multicalibration properties of ERM. We will include this discussion in the revision.

Q4: While results differ in minor ways between the different tabular datasets, we observe that models which are calibrated “out of the box” tend to not benefit from multicalibration post-processing in a statistically meaningful way (Observation 1). On four of the five tabular datasets, looking only at the max-smECE metric of MLP ERM, the standard deviations overlap with the opposing mean (MEPS, ACS Income), or (mean ± sd) values come within 0.003 of each other (Bank Marketing, Credit Default). As you point out, on HMDA (the remaining fifth dataset), max-smECE of MLP ERM is improved drastically by post-processing. On HMDA, however, MLP ERM is poorly calibrated to begin with: relative to the smECE of MLP HKRR (0.005 ± 0.001), or even the smECE of MLP Temp (0.022 ± 0.006), the smECE of MLP ERM (0.049 ± 0.006) is quite large. Therefore, observation 1 does not apply (as strongly) to the MLPs trained on HMDA. On the four other tabular datasets, MLP ERM achieves smECE much closer to that of the post-processed MLPs. Outside of MLPs, we find that miscalibrated models (e.g. Naive Bayes, SVMs) tend to benefit from multicalibration post-processing throughout our tabular experiments (Observation 2). We can provide specific examples if necessary (omitted here due to space).

Q6: Note that an increase in calibration fraction implies less data is used to train the base predictor, and more data saved for post-processing. As you point out, max smECE increases as we increase the calibration fraction in Fig 3. This is likely since the base predictor has poorer generalization when trained on less data, and this performance is not rectified by multicalibration post-processing. Please also see Q3.

We thank the reviewer for their time and thoughtful feedback. If the pressing concerns have been sufficiently addressed, we kindly ask the reviewer to consider adjusting their score by taking the rebuttal into account. Thank you!

ACSIncome Results: initial

Results: new

Takeaways:

Old groups: (Observation 1) Multicalibration does not improve upon ERM in a statistically significant way. Note here also the poor performance in accuracy of HKRR in the results included in the paper. This is because we selected the best performing hyperparameter based on worst group smECE, which does not prioritize accuracy. However, there is another choice of hyperparameters which performs slightly worse for worst group smECE on the validation (and test) set, but achieves competitive accuracy of around 0.80. Furthermore, this relates to tradeoffs discussed in Observation 3. We will note this in the next draft as well.

New groups: Multicalibration does not improve upon ERM + Platt scaling in a statistically significant way (Observation 1 and 2).

As requested, we ran some additional experiments for MLPs on the tabular datasets with different --- and in some cases, more complex --- groups. We present a few of the experiments here, and echo takeaways relative to the old experiments (with different group definitions). Unfortunately, we can no longer submit a PDF anywhere, so have included the results as markdown tables. Due to character limits, these tables may have to be split among multiple respones.

Overall, the takeaway for MLPs largely remains the same: it is difficult to find instances where multicalibration helps in a statistically significant way over ERM or some form of simple calibration (Observations 1 and 2). Further, where multicalibration does help, this help sometimes comes at a cost to accuracy (Observation 3).

For each dataset, we include the "initial" results which are in our submitted draft, and new results which we ran at the reviewers requests.

ACS Income

Group definitions: initial

Group definitions: new

Bank Marketing Results: initial

Results: new

Takeaways:

Old group definitions: (Observation 1 and 2) Multicalibration does not outperform Isotonic / Temperature scaling in a statistically significant way.

New: (Observation 1 and 2) Multicalibration does not outperform ERM or Temperature scaling in a statistically significant way.

Credit Default Results: initial

Results: new

Takeaways

Old: (Observation 3) Statistically significant improvement of HKRR to worst group smECE over ERM and calibration. However, comes at a cost to accuracy. HJZ sees no significant improvements.

New: Unchanged

MEPS Results: initial

Results: new

Takeaways

Old: (Observation 1) No statistically significant improvement by multicalibration algorithms.

New: Unchanged.