A Closer Look at AUROC and AUPRC under Class Imbalance

Thank you for your apt and constructive review!

W1: Can the presentation be improved, particularly regarding the clarity of "prevalence"?

In essence, in Theorem 3, we show that if one group of samples has a much higher outcome rate (prevalence) than another (e.g., men are more likely than women to receive a correct diagnosis for a heart attack), optimizing for AUPRC will provably preferentially optimize to fix mistakes that affect the high prevalence group over the low prevalence group (e.g., the model will preferentially learn to identify heart attack symptoms for men at the expense of learning those for women). In settings where such a model is used to determine who should receive some limited resource (e.g., to be evaluated by a cardiac specialist after an ED visit), this translates into a disparity in resource allocation between the two groups, which in many cases may be undesirable.

To clarify this in our text, we have added: "Essentially, Theorem 3 (proof provided in Appendix F) shows the following. Suppose we are training a model $f$ over a dataset with two subpopulations: Population $a=0$ and $a=1$. If the model $f$ is calibrated and the rate at which $y=1$ for population $a=0$ is sufficiently low relative to the rate at which $y=1$ for population $a=1$, then the mistake that, were it fixed, would maximally improve the AUPRC of $f$ will be a mistake purely in population $a=1$. This demonstrates that AUPRC provably favors higher prevalence subpopulations (those with a higher base rate at which $y=1$) under sufficiently severe prevalence imbalance between subpopulations."

This clarifies that prevalence relates to the probability of $y=1$ for a specific subgroup. The prevalence disparity is captured by the limit as $P(y=1|a=0)$ approaches zero while $P(y=1|a=1)$ remains fixed. If you have further feedback on how to further clarify this point, please let us know.

W2: How does your work relate to the NeurIPS 2022 paper on AUPRC optimization?

Thank you for pointing out this related work! We've included it as a reference and explained how our findings are synergistic with and extend upon this excellent prior work:

1. In our proof of Theorem 1: "Note that this formulation of AUPRC reflects earlier, different formulations of AUPRC, such as those found in the AUPRC optimization literature (Wen et al., 2022)"
2. In our Synthetic Results section, where we comment on the impact of optimizing for AUROC vs. AUPRC: "These results demonstrate explicitly that not only does optimizing for AUPRC differ greatly than for AUROC, as has been noted historically by researchers developing explicit AUPRC optimization schemes (Wen et al., 2022), but it in fact does in an explicitly discriminatory way in very realistic scenarios."
3. Finally, in Section 5 (our literature review), we note that "The widespread nature of Claim 1 has also led researchers astray when exploring new optimization procedures for AUPRC, by advocating for the importance of AUPRC when processing skewed data, even in domains such as medical diagnoses that often have high false negative costs relative to false positive costs (Wen et al., 2022).""

To further clarify the novelty of our work relative to that of Wen et al., note that while Wen et al. provide a related probabilistic expression for AUPRC designed to facilitate their optimization algorithm, our Theorem 1 (which presents a different formulation of AUPRC) is designed to clearly show the mathematical relationship between AUROC and AUPRC. Our impact comes from realizing the implications of this relationship on the strengths and weaknesses of these metrics, challenging the popular opinion that class imbalance is the defining factor in their distinct use cases.

W3: Can you shorten the caption of Figure 1?

We've shortened the caption considerably; it now reads: "a) Consider a model $f$ yielding continuous output scores for a binary classification task applied to a dataset consisting of two distinct subpopulations, $\mathcal{A} \in \{0, 1\}$. If we order samples in ascending order of output score, each misordered pair of samples (e.g., mistakes 1-4) represents an opportunity for model improvement. Theorem 3 shows that a model's AUROC will improve by the same amount no matter which mistake you fix, while the model's AUPRC will improve by an amount correlated with the score of the sample. b) When comparing models absent a specific deployment scenario, we have no reason to value improving one mistake over another, and model evaluation metrics should therefore improve equally regardless of which mistake is corrected. c) When false negatives have a high cost relative to false positives, evaluation metrics should favor mistakes that have lower scores, regardless of any class imbalance. d) When limited resources will be distributed among a population according to model score, in a manner that requires certain subpopulations to all be offered commensurate possible benefit from the intervention for ethical reasons, evaluation metrics should prioritize the importance of within-group, high-score mistakes such that the highest risk members of all subgroups receive interventions. e) When false positives are expensive relative to false negatives and there are no fairness concerns, evaluation metrics should favor model improvements in decreasing order with score."

Q1: Can you provide analysis based on multiple subgroups?

This is a great question and a rich area for future work. We've added to our Future Work section: "Firstly, our theoretical findings can be refined and generalized to... specifically take into account more than 2 subpopulations for more nuanced comparisons beyond what can be inferred through pairwise comparisons between subpopulations, where our results would naturally apply".

Thank you for your comprehensive review and valuable comments!

W1: Can you reduce the repetition in Section 4 and Figure 1?

Thank you for this suggestion! We've condensed Figure 1's caption and Section 4. However, we maintain some overlap to ensure clear presentation of our key takeaway: understanding when to prefer AUROC or AUPRC as optimization metrics. To demonstrate these changes within the space limitations of the "official rebuttal", we provide our updated Figure 1 caption in response to your concern specifically about its redundancy later in this rebuttal.

W2: Why does Figure 2 appear corrupted?

We do not see any issues with Figure 2 on OpenReview, using Google Chrome's built-in PDF viewer. Could you please provide more details about the issue and your PDF viewer/OS in a comment? We will follow up promptly to ensure all figures are properly rendered.

W3: Can you clarify the explanation in lines 129-133?

In essence, we show that if one group of samples has a much higher rate of the outcome than another (e.g., men are more likely than women to receive a correct diagnosis for a heart attack), optimizing for AUPRC will provably preferentially optimize to fix mistakes that affect the high prevalence group over the low prevalence group (e.g., the model will preferentially learn to identify heart attack symptoms for men at the expense of learning those for women). In settings where such a model is used to determine who should receive some limited resource (e.g., to be evaluated by a cardiac specialist after an ED visit), this preferential optimization procedure will translate into a disparity in resource allocation between the two groups, which in many cases may be undesirable.

In the text we have reworded this key result: "Essentially, Theorem 3 (proof provided in Appendix F) shows the following. Suppose we are training a model $f$ over a dataset with two subpopulations: Population $a=0$ and $a=1$. If the model $f$ is calibrated and the rate at which $y=1$ for population $a=0$ is sufficiently low relative to the rate at which $y=1$ for population $a=1$, then the mistake that, were it fixed, would maximally improve the AUPRC of $f$ will be a mistake purely in population $a=1$. This demonstrates that AUPRC provably favors higher prevalence subpopulations (those with a higher base rate at which $y=1$) under sufficiently severe prevalence imbalance between subpopulations."

W4: Can you avoid using boldface and italics simultaneously?

Thank you for this suggestion. We've removed all such instances.

W5: Can you remove self-praising expressions?

Thank you for this apt suggestion! We have removed all such instances.

W6: Can you reduce redundancy in Figure 1's caption?

Great suggestion. We have removed the repeated definition of the mistake from the figure caption. The whole caption of Figure 1 (which has been further condensed) is shown below:

"a) Consider a model $f$ yielding continuous output scores for a binary classification task applied to a dataset consisting of two distinct subpopulations, $\mathcal{A} \in \{0, 1\}$. If we order samples in ascending order of output score, each misordered pair of samples (e.g., mistakes 1-4) represents an opportunity for model improvement. Theorem 3 shows that a model's AUROC will improve by the same amount no matter which mistake you fix, while the model's AUPRC will improve by an amount correlated with the score of the sample. b) When comparing models absent a specific deployment scenario, we have no reason to value improving one mistake over another, and model evaluation metrics should therefore improve equally regardless of which mistake is corrected. c) When false negatives have a high cost relative to false positives, evaluation metrics should favor mistakes that have lower scores, regardless of any class imbalance. d) When limited resources will be distributed among a population according to model score, in a manner that requires certain subpopulations to all be offered commensurate possible benefit from the intervention for ethical reasons, evaluation metrics should prioritize the importance of within-group, high-score mistakes such that the highest risk members of all subgroups receive interventions. e) When false positives are expensive relative to false negatives and there are no fairness concerns, evaluation metrics should favor model improvements in decreasing order with score."

W7: How do you address the strong assumptions in theoretical results, e.g., perfect calibration?

This is true; We acknowledge this limitation and have extended our discussion in Section 6: "In addition, one of the largest limitations of Theorem 3 is its restrictive assumptions, in particular the requirement of perfect calibration. A ripe area of future work is thus to investigate how we can soften our analyses for models with imperfect calibration or to determine whether or not our results imply anything about the viability or safety of post-hoc calibration of models optimized either through AUPRC or AUROC"

W8: Can you fix the typos on lines 290 and 310?

Thank you for catching these! We have corrected both.

Q1: Have you considered models with ascending levels of calibration?

This is a great area for future work. We believe it's possible but requires careful specification of model miscalibration. For example, bounding a model's calibration error over any region of the output score space could help bound the extent to which prevalence gaps correspond to different prediction rates, translating into bounds on AUPRC's preference for fixing high-prevalence group mistakes. We've highlighted this in Section 6 with strengthened language.

Thank you for your time, expertise, recognition of the impact of our work, and helpful suggestions. Below, we address each of your questions or concerns individually.

W1: Why is the variance so high in the experiments?

Two factors contribute:

1. We report confidence intervals spanning the 5th to 95th percentiles, not standard errors of the mean; these confidence intervals will, therefore, remain wide no matter how many samples we run and not shrink towards the mean as some other measures of variance do.
2. Our procedure samples from an extremely wide space of possible models, resulting in high true variance by design. This is important so we can make sufficiently general conclusions about the validity of our theory.

We've added the following to Figure 2's caption to help clarify: "Synthetic experiment per-group AUROC, showing a confidence interval spanning the 5th to 95th percentile of results observed across all seeds" to help clarify this.

For the real-world data, these experiments are actually somewhat expensive to run, as we sample a very large number of models, seeds, and hyperparameter options in order to sufficiently assess the correlations of the fairness disparities under AUROC and AUPRC, so using larger datasets does pose an additional challenge there as well. Relatedly, the datasets we have chosen are some of the more common fairness datasets, so without branching into much more intensive modeling domains, we have limited options for additional datasets that are widely used.

That said, your point is still well taken and we will explore other ways to present these results in our revision to better demonstrate the consistency and statistical reliability of aggregate results across many samples, while still communicating the raw variance.

Q1: There is a mistake on line 199/200.

Thank you for catching this typo! We have corrected this sentence to be "We also evaluate the test set AUROC gap between groups, where gaps are defined as the value of the metric for the higher prevalence group minus the value for the lower prevalence group."

Q2: What Algorithmic Procedure Obtains Scores at the Target AUROC?

We outline the procedure in Appendix G.1. Briefly, we randomly sample scores for positively labeled samples, then draw scores for negatively labeled samples under a distribution that ensures, in expectation, the count of positive samples with scores above or below any given negative sample will be precisely the target AUROC. While exact for single-group AUROC, it is not necessarily exact for multi-group AUROC constraints, though it works well in practice.

To improve clarity, we have added the following text to our paper:

1. On line 154, the sentence now reads "... such that $\text{AUROC}(\mathcal D_1) \approx \text{AUROC}(\mathcal D_2) \approx \text{AUROC}(\mathcal D_1 \cup \mathcal D_2) = 0.85$ (See Appendix G.1 for technical details; ...)."
2. In Appendix G.1, we have added a new paragraph at the end which states "The procedure outlined above guarantees that, in expectation, the AUROC of the generated set of scores and labels will be precisely the target AUROC. However, if you apply this procedure independently across different sample subpopulations, this guarantee can only be applied on each subpopulation individually, and not necessarily on the overall population due to the unspecified xAUC term. However, in practice, for the experiments we ran here, that impact neither meaningfully impacts our experiments nor were the joint AUROCs sufficiently different from the target AUROC to warrant a more complex methodology."

Q3: What is the precise reordering procedure applied?

This procedure is described in detail in Appendix G.3 of the submitted Manuscript, subsection "M3. Sequentially Permuting Nearby Scores." In short, the reordering procedure selects a random permutation of scores, realized as a permutation matrix, such that that permutation matrix has no entries more than 3 positions off the diagonal (thus ensuring that samples cannot change position by more than 3 places). We have amended line 163 to reference this, stating "... (sorted) model scores up to 3 positions (See Appendix G.3 for details)." The implementation of this procedure is released along with the rest of our experimental code at the link included in the manuscript. If requested, we are happy to add more details about the procedure we use to the text or the comments here.

Thank you for your many insightful and helpful comments to improve our work! Note that, for space reasons, while we have responded to all your identified weaknesses individually in our official "rebuttal" (which may or may not be released to reviewers at the time this comment is visible, depending on openreview settings), in this "comment" we respond to your questions instead. Please see the rebuttal as well for our full response to your excellent review.

Q1: What causes the high variance in Figure 2?

Two factors contribute:

1. We report confidence intervals spanning the 5th to 95th percentiles, not standard errors of the mean; these confidence intervals will therefore remain wide no matter how many samples we run, and not shrink towards the mean as other measures of variance do.
2. Our procedure samples from an extremely wide space of possible "models" resulting in high true variance by design. This is important so we can make sufficiently general conclusions about the validity of our theory.

We've added to Figure 2's caption: "Synthetic experiment per-group AUROC, showing a confidence interval spanning the 5th to 95th percentile of results observed across all seeds" to help clarify this.

Q2: What do 'Prevalence (Higher)' and 'Prevalence (Lower)' mean?

These refer to the fraction of samples with label 1 in the subpopulations with the highest and lowest prevalence, respectively. They don't sum to 1 as they're for different subpopulations. E.g., if we are predicting the likelihood a patient has had a heart attack based on their symptoms for a population containing both men and women, both subgroups will have different rates of heart attacks, and those rates will not sum to one.

To clarify, we've added to Table 1's caption: "Here, Prevalence (Higher)'' refers to the rate at which the prediction label $y=1$ for the subpopulation with a higher such rate, and Prevalence (Lower)'' refers to the same rate but over the subpopulation of the dataset with a lower rate of $y=1$."

Thank you for your insightful and comprehensive review! Note that, for space reasons, we respond to your two questions in a "comment" rather than here in our official "rebuttal."

W1: Why optimize for AUPRC?

AUPRC is implicitly used as an optimization metric in cases where it is the target for hyperparameter tuning or model selection (as we explore in our real-world experiments). It also can be an explicit optimization target via specialized algorithms (e.g., here, here).

Our work shows that optimizing for AUPRC favors subpopulations with higher outcome base rates. We've added clarification to Section 3: "Simulating optimizing by these metrics allows us to explicitly assess how the use of either AUPRC or AUROC as an evaluation metric in model selection processes such as hyperparameter tuning can translate into model-induced inequities in dangerous ways."

W2: Why not use boosted trees for the synthetic example?

While in our real-world experiments, we use boosted trees for exactly the reasons you highlight, in our synthetic experiments we need to use more controlled optimization procedures so that we can precisely attribute the results to the choice of optimizing via AUROC or AUPRC in isolation. In particular, using a classical model risks confounding the impact of AUROC vs. AUPRC with factors such as the ease or difficulty of optimizing towards the target for various subpopulations, optimization algorithms, or output score regions.

To make this clearer, we have added to Section 3.1: "In this section, we use a carefully constructed synthetic optimization procedure to demonstrate that, when all other factors are equal, optimizing by or performing model selection on the basis of AUPRC vs. AUROC risks exacerbating algorithmic disparities in the manner predicted by Theorem 3. For analyses under more realistic conditions with more standard models, see our real-world experiments in Section 3.2."

W3: What does Theorem 3 add beyond the definition of AUPRC?

Our work's theoretical analyses provide several key insights:

1. Theorem 1 demonstrates that AUPRC & AUROC can be expressed as very similar linear functions of the expectation of the model's FPR over positive class scores. This reveals that the key difference between AUROC & AUPRC is not precisely that AUPRC focuses more on the positive class or doesn't care about true negatives, but rather that AUPRC weights model errors more heavily when they occur in high-score output regions vs. low-score regions, whereas AUROC weights all errors equally.
2. Theorem 3 formalizes the intuition that you rightly identify: that given AUPRC's focus on the high-score region, subpopulations with a higher prevalence of the label will be preferentially optimized by AUPRC. To the best of our knowledge, this has not been previously proven formally.
3. Critically, these theoretical formulations demonstrate both the widespread misconception in the ML community that AUPRC is generally superior for skewed data (Section 5) and formally prove why this misconception is dangerous to model fairness. In particular, despite the intuitive recognition that some top experts like you may have about AUPRC's fairness risks, many authors have used AUPRC as a selection and principal evaluation metric in fairness-centric settings, as noted in a variety of the papers in Section 5.

W4: Doesn't the mistake model favor AUROC over AUPRC?

We agree that correcting adjacent pair mistakes can have varying impacts on AUPRC depending on the threshold. This behavior--and critically, its consequences on model fairness and appropriate use cases--is precisely what we want to highlight to readers in our work. In particular, this model makes it clear that in high-score regions and retrieval contexts without fairness constraints, AUPRC is appropriate. For optimization problems that heavily depend on lower-score regions, AUPRC is less appropriate. This holds regardless of class imbalance, despite the widespread belief in the community that AUPRC should be generally preferred in cases of class imbalance.

W5: Can you provide actual AUROC and AUPRC values?

Yes, we will make these results available in our revision. However, we have two notes about these raw values:

1. Our experimental analyses are focused on the impact of performing model selection or other optimization by AUROC vs. AUPRC under varying levels of prevalence imbalance. This question is therefore inherently not about the precise AUROC and AUPRC disparities in a single model run, but rather about how optimizing by AUROC vs. by AUPRC impacts fairness gaps in aggregate across many hyperparameter settings and prevalence disparities.
2. Accordingly, the number of "actual AUROC and AUPRC values" we have is quite large. For our experiments, we train a very large number of models (500+) over different hyperparameters and across many seeds so that we can assess the impact of the evaluation choice with sufficient statistical power. Nevertheless, we will absolutely make all these results available in our revised manuscript.

W6: Shouldn't readers explore a variety of metrics?

You're right; to clarify this, we have added the following to the top of Section 4: "Note that while we provide guidance below on situations in which AUROC vs. AUPRC is more or less favorable, this is not to suggest that authors should not report both metrics, or even larger sets of metrics or more nuanced analyses such as ROC or PR curves; rather this section is intended to offer guidance on what metrics should be seen as more or less appropriate for use in things like model selection, hyperparameter tuning, or being highlighted as the 'critical' metric in a given problem scenario."

Thank you again for your time and valuable feedback! As the rebuttal period comes to a close, we were wondering if our response has adequately addressed your concerns. If there are any remaining questions or comments, we would be happy to discuss!