Overcoming Common Flaws in the Evaluation of Selective Classification Systems

Thank you again for your valuable comments, and for taking the time to read our general reply, as well as considering our point-by-point comments here:

W1. “Fig. 4: the risk vs. coverage curves for AURC vs. AUGRC do not look that different. Sure, there is non-monotonicity in AURC, but that is in the low-coverage region, which presumably is not of practical interest. The rest looks monotonic and fairly similar. It is also true that the two metrics suggest different CSFs, but again, they aren't that different. To be clear, I still think AUGRC is favorable, just saying that the difference appears modest.”

• While the overall shape of the Selective Risk-Coverage is indeed similar to that of the Generalized Risk-Coverage curve, we demonstrate in our experiments that even smaller visual changes in the curves lead to alterations in method rankings and are thus highly relevant.
• Further, we want to point out that the low-coverage region may also be relevant in practice, for instance considering a clinical deferral-system where a large fraction of cases may be deferred to ensure very high accuracy on the accepted cases. When evaluating specific applications for which certain risk or coverage intervals are known to be irrelevant, adaptations such as a partial AUGRC (analogous to the partial AUROC) may be considered. In response to your feedback, we added the note on partial evaluation of the AUGRC to the conclusion section.

W2. “I think the key conclusion, which is the recommendation to adopt AUGRC, is valid, however the language in the section is a bit too strong (there is "substantial" twice and "significant limitations"). I recommend to tone-it-down.”

• Thank you for this helpful comment. In the updated manuscript, we toned down the conclusion, as you proposed, removing the word “significant” in “the current metrics have significant limitations” (line 276) and “substantial” in “substantial deviations from intended and intuitive performance assessment behaviors” (line 282-283).

Q1. “The authors claim that Selective Classification is a crucial components for clinical diagnostics ML systems (among other application domains). Can you provide examples of classifiers in clinical use that incorporate Selective Classification?”

• Three interesting examples would be the following:
  • Dvijotham et al., “Enhancing the reliability and accuracy of AI-enabled diagnosis via complementarity-driven deferral to clinicians”: The authors introduce the SC-based CoDoC deferral system for breast cancer screening and TB triaging, showing “that combined AI-clinician performance using CoDoC exceeds that currently possible through either AI or clinicians alone”.
  • Leibig et al., “Combining the strengths of radiologists and AI for breast cancer screening: a retrospective analysis”: The authors propose an SC-based decision-referral approach for breast-cancer screening based on mammography data.
  • Bungert et al., “Understanding Silent Failures in Medical Image Classification”: The authors comprehensively evaluate SC in the biomedical field and develop an interactive tool that facilitates identifying silent failures.

Q2. “What is AUROC_f? It is introduced on line 172, however the description is unclear and there are no references.”

• The “failure AUROC” AUROC$_f$ is the standard AUROC but computed on the binary failure labels (correctly classified vs. failure, denoted $Y_f$ in this paper) and the confidence scores. We added a corresponding reference in Section 2.4.

Thank you for your constructive feedback. As we believe in having resolved your comments, please let us know in case there are remaining concerns that would prevent you from recommending a stronger acceptance of our work.

Thank you again for your valuable comments, and for taking the time to read our general reply, as well as considering our point-by-point comments here:

W1. “The empirical evaluation can be improved.”

W1.1. “there are some contradictory lines [...]”

• Thank you for pointing out the mistakes in Section 4.2. It should read “erroneously favors MCD-PE over DG-RES” in line 268-269, and it should read “favoring MCD-PE [...] compared to DG-Res” in the caption of Figure 4. We corrected the sentences in the manuscript.

W1.2. “the authors never specify whether they correct for multiple outcomes testing [...]”

• Thank you for pointing this out. Please find our detailed response in the general response to all reviewers above.

W2. “[...] straigthforward interpretation of the AUGRC score in this context (lines 206-208).”

• Arguably most intuitively, the AUGRC corresponds to the rate of silent failures averaged over working points. In the suggested context, the interpretation is: On drawing two random samples, it corresponds to the probability that either both are failure cases or that one is a failure and has a higher confidence score than the non-failure. This can be read from Equation 7: $AUGRC = (1 − AUROC_f ) · acc · (1 − acc) + (1 − acc)^2 / 2$ The second term is the probability of drawing two failures. The first term is the probability that one is a failure times the probability that the failure is ranked higher (1 − AUROC$_f$). As an example, a CSF that outputs random scores yields AUROC$_f=0.5$, and hence AUGRC $=(1-acc)/2$.
• In response to your feedback, we changed “expected risk of silent failures across working points” in lines 201-202 to “rate of silent failures averaged over working points”. Further, we extended the explanation on Equation 7 to include the AUROC-analogous interpretation as well as the example case of a random CSF.

W3. “[...] the paper heavily relies on acronyms”

• Thank you for raising this point. In response to your feedback, we reintroduced the abbreviations at important places in the paper (e.g. beginning of Section 2) or used the full term instead. We further updated Section 2.4 to introduce all acronyms with the full term for improved readability.

W4. “[...] a brief discussion regarding the computational time required to compute AURGC should be included in the main paper.”

• Thank you for bringing up this point. The main bottleneck of the AUGRC computation is the sort operation on the confidence scores (O(n log n)). This is the same as for the AURC and there is no computational overhead as compared to the AURC.
• Benchmarking the computational time for evaluating the AURC, AUGRC, and AUROC$_f$ for 1k random scores and failure labels, we obtain the following results:
  AUROC$_f$: 730μs ± 13μs (sklearn implementation)
  AURC: 564μs ± 2μs (ours)
  AUGRC: 562μs ± 2μs (ours)
• In response to your feedback, we added a note on the AURC computation time in the beginning of Section 4, and details and benchmark results to a new section in the Appendix: “A.2.3 Computation Time”.

Q1. & Q2. Addressed in W1-W3.

Q3. “[...] confidence score is always zero, is the AUGRC 0 [...]? If so, is AUGRC favoring classifiers that tend to be underconfident and asbtain too much? [...]”

• To clarify, whether a SC model abstains given a predicted confidence score depends on a threshold which is fixed on application. However, a multi-threshold evaluation, like in AUGRC, does not involve a fixed threshold but aggregates across all working points. More precisely, no explicit abstention decision is made in the AUGRC, the same as no classification decision is made in the classification AUROC.
• Illustrating the latter with an example: A binary classifier that always outputs $P(Y=1)=0$ will yield an AUROC of ½ being equivalent to the random classifier. Analogously, if the confidence scores are always zero, the AUROC$_f$ is ½ and following Equation 7 the AUGRC is $(1-acc)/2$. Thus, the AUGRC does not favor underconfident classifiers.
• We extended the explanation on Equation 7 (see our response to W2).

Q4. “[...] why do not the authors consider the classification quality [...]? [...]”

• [2] define the Classification quality (Q) as the fraction of rejected failures plus the fraction of non-rejected non-failures.
• Using 1-Q as a replacement of Generalized Risk in the AUGRC deviates from the intention of evaluating the silent failure risk, as this assigns a loss of 1 to rejected but correctly classified samples. As an example of resulting unintended behavior, two classifiers with accuracy 0 and 1, respectively, both yield a Q-based AUGRC of ½. This breaks requirement R2.
• Please note that instead of a holistic evaluation suitable for method benchmarking, [2] derives “performance measures [...] [that] correspond to a reference operating point”. Thus, Q may be used for working point selection but is not suitable for replacing Generalized Risk in the AUGRC.
• We added to Section 2.1: “Aside from the selective risk, other performance measurements for working point selection include for example Classification quality and Rejection quality [2]”.

[2] Condessa et al., “Performance measures for classification systems with rejection”

Thank you for your constructive feedback. As we believe to have resolved your comments, please let us know if there are any remaining concerns that would hinder a recommendation for acceptance.

Thank you again for your valuable comments, and for taking the time to read our general reply, as well as considering our point-by-point comments here:

W1. “Limited discussion of future works.”

• Thank you for this helpful comment. In response to your feedback, we extended the Conclusion Section to point out directions of future work: Our proposed evaluation framework provides a solid basis for future work in Selective Classification, including developing novel methods as well as analyzing the properties of individual methods.

W2. “NLL and Brier Score are listed among the multi-threshold metrics, but there's no thresholding involved in their evaluation. In fact, it should be possible to use them as risk measures, which would cover more for than just 0/1 loss, especially where probabilities are important for decision making.”

• Thank you for raising this point. Indeed, the NLL and Brier Score do not involve thresholding of confidence scores. In response to your feedback, we rephrased line 167 to: “Importantly, proper scoring rules such as the Negative-Log-Likelihood and the Brier Score are technically not multi-threshold metrics. Yet, we include them here as they also aim for a holistic performance assessment, i.e. assessment beyond individual working points.”
• Regarding your proposal to use NLL or Brier Score as risk measure: NLL and BS both assess the calibration of confidence scores in addition to their ranking. Generally, the calibration task can be viewed as orthogonal to that of Selective Classification where rejection is solely based on the ranking of scores. For a detailed discussion on the relation between calibration and confidence ranking we refer to Appendix C in [1]. Further investigating the relation between confidence ranking and calibration and on evaluating SC in settings where calibrated scores are of interest is an interesting direction of future work, which we added to the conclusion section in the updated manuscript.

[1] Jäger et al., “A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification”

Q1. “A similar task to SC is classification with rejection of OOD samples (sometimes called open-set recognition). In such cases, there isn't a ground truth to evaluate the risk on (in practice). Can the authors envision a way to adapt their metric to these scenarios?”

• We understand you are referring to settings where the SC model is validated on data containing classes that were not part of the training data. The AUGRC metric based on the binary failure label is directly applicable to these scenarios, as any prediction on unknown classes is per definition a misclassification (for a visualization, see Figure 5 in [1]). In Table 3, we report AUGRC values evaluated on various distribution shifts, also containing semantic and non-semantic new-class shifts.
• In case you are referring to a different kind of open-set recognition, we would ask for a more concrete task formulation for us to comment on.

Q2. “The metrics were scaled by ×1000 in Figure 2. Were they also rescaled for all the other results?”

• Thank you for pointing out the missing information. We rescaled all results apart from Figure 5. More precisely, we scaled the AURC and AUGRC values by 1000 in Figure 4 and Table 3. For the color-coded AUGRC in Figure 5, no rescaling was done. In response to your feedback, we updated the caption of Figure 4 to clearly state which values were rescaled.

Q3. “What does the optimal AUGRC mean? What would be the AUGRC of the Bayes-optimal model for a task? Is there an AUGRC value associated to a random classifier, as in AUROC?”

• The optimal AUGRC refers to the AUGRC for an optimal CSF, i.e. one that ranks all failure cases lower than correctly classified cases (AUROC$_f = 1$), but is conditioned on the performance of the underlying classifier. Please note that the CSF is not directly linked to the classifier output, but can be any (external) function that provides continuous confidence scores per classified sample. As such, in the general formulation, classifier and CSF are two independent entities and AUGRC assesses the performance of both jointly.
• The Bayes-optimal classifier by itself does not directly define a CSF and hence does not correspond to a specific AUGRC value. We believe that investigating the relation between Bayes-optimality and SC is an interesting direction of future work.
• On the level of detecting failure cases, i.e. given a fixed classifier with accuracy $acc$, assigning random confidence scores yields a AUROC$_f=0.5$. Following Equation 8, this corresponds to AUGRC $=(1-acc)/2$. The same holds if the classification itself is random. There, however, the accuracy depends on the number of classes $K$, i.e. $acc=1/K$.
• In response to your questions, we re-formulated line 206 to: “The AUGRC for an optimal CSF (AUROC$_f=1$) is given by the second term in Equation 7”. We also added the AUGRC of the random classifier to the same section.

Q4. “Any intuitions about why the rankings of AURC and AUGRC where the same for iWildCam?”

• For the iWildCam dataset, we observe very robust method rankings for both AURC and AUGRC which don’t change between the two metrics. Overall, the volatility of method rankings varies from dataset to dataset (Figure 3).
• We agree that the investigation of specific methods and datasets is an interesting direction of future work, which we added to the conclusion section in the updated manuscript.

Thank you once more for your constructive feedback. As we believe to have resolved your comments, please let us know in case you have remaining suggestions to further increase the quality of our work.

Thank you again for your valuable comments, and for taking the time to read our general reply, as well as considering our point-by-point comments here:

W1. “It is important to add the details of the methods in the main paper. For example, DeepGamblers (DG) is referred to as DG in the main paper and this abbreviation is only explained in the appendix. Similarly for other baselines.”

• Thank you for pointing out that the abbreviations DG and MCD-PE were used in the main text without reference to the explanation in the appendix. This was particularly the case in Section 4.2, where we explicitly compare these two confidence scoring functions. In response to your feedback, we adapted Section 4.2 to properly introduce the abbreviations DG and MCD-PE.
• While the main paper is now fully self-contained and all abbreviations are introduced, we would like to comment on the general decision to provide the details of utilized confidence scoring functions in the appendix. This is because the focus of our work is not on the confidence methods like in other studies, but on the metric, which is why we find it important to allocate sufficient space in the main paper to metric-related descriptions.

W2. “Although the experiments are extensive, it is good to include the recent state-of the art SC methods such as Feng et al (2023) to understand which method is the best to use.”

• Please note that the focus of our experiments is to show the relevance of the proposed metric rather than the performance of individual methods. We argue that this metric relevance can be demonstrated with a diverse and representative set of prevalent confidence scoring functions, but it is not necessary to include all existing methods.
• The main conclusion in [2] is that “[...] selecting according to classification scores is the SOTA selection mechanism for comparison” (based on experiments with three SC methods, including DG). We want to point out that with DG-MCD-MSR, DG-PE, and DG-TEMP-MLS, our empirical evaluation already includes multiple CSFs that are trained with DG loss attenuation but are based on the classifier’s logits, aligning with the recommendation in [2].
• We agree that testing and comparing further novel methods with our proposed AUGRC metric is an interesting direction for future work. However, running all experiments required for this method within one week is not feasible for us. In response to your feedback, we extended the first paragraph in Section 4 to include the method of Feng et al. in the discussion about future work.

[2] Feng et al., “Towards better Selective Classification”

Thank you once more for your constructive feedback. As we believe to have resolved your comments, please let us know in case you have remaining suggestions to further increase the quality of our work.