What Does It Take to Build a Performant Selective Classifier?

We thank the reviewer for their in-depth feedback, and for highlighting the clarity, practical value, and balance of our paper. The reviewer raises important points about the presentation of the oracle bound, the calibration discussion, and the need for clearer exposition and empirical support in parts. We address each of these concerns below and have revised the paper to incorporate the suggested corrections.

---

1) Missing nuance in the presentation of the oracle.

Definition 3 assumes no Bayes noise.

We agree with the reviewer that this assumption was indeed not stated explicitly. We have revised the sentence immediately following Definition 3:

Assuming no Bayes noise—that is, all errors are avoidable given perfect confidence—this piecewise curve (see Figure 1) traces the oracle accuracy–coverage frontier based on a perfect ranking of examples by correctness probability.

What kind of oracle?

After rephrasing the previous sentence, this distinction becomes immediate under the no-Bayes-noise assumption: when every example’s posterior correctness $\Pr\bigl(h(x)=Y \mid x\bigr)\in\{0,1\}$, ranking by the Bayes-optimal posterior is identical to ranking by the true realized label. Hence, in the no-Bayes-noise setting the two oracle interpretations coincide.

State that lower approximation error also raises the upper bound.

As Definition 3 makes explicit, the oracle curve $\overline{\mathrm{acc}}(a_{\text{full}}, c)$ depends directly on the full-coverage accuracy $a_{\text{full}}$. Thus, improving approximation error (i.e., raising $a_{\text{full}}$) not only reduces the selective classification gap $\Delta(c)$, but also lifts the upper bound itself across all coverage levels. This interaction is indeed visible in Figure 2 but was indeed not emphasized in the text. We have now clarified this point directly after Definition 4 and revised the figure caption to highlight how changes in approximation accuracy affect both the bound and the observed gap.

---

2) Overstated contribution on monotone calibration.

We thank the reviewer for pointing out this issue and have revised the draft to clarify our intent. While it is well understood that monotone transforms preserve score ordering and therefore cannot improve ranking-based risk–coverage curves, this point is often muddled in the literature—especially given empirical studies [1, 2] that appear to show large selective-classification gains from temperature scaling. Our contribution is to (i) formalize this invariance result in the multi-class case, (ii) identify rare logit-margin ties as the only way monotonicity can fail to preserve ranking, and (iii) highlight that performance improvements must stem from calibration methods that actively rerank examples. We now present this section as a clarification aimed at resolving apparent contradictions in prior work, not as a standalone result. We hope this helps readers more clearly distinguish between calibration techniques that change confidence scores and those that change rank order.

---

3) Space for improvement presenting Eqs. 9-12

Clarify that Eq. 10 keeps the selection function constant.

Thank you for pointing out the potential ambiguity. To make the role of each term crystal-clear, we have added an additional clarification on this point after Eq. (9)-(12) that explicitly states:

Here we freeze the acceptance set $A_c$ defined by the current scoring function and ask how much worse the learned classifier $h$ is than the Bayes-optimal rule on this same set.

We then remind the reader that any shifts in the acceptance region caused by replacing $h$ are not charged to $\varepsilon_{\text{approx}}(c)$ but are instead captured by the ranking-error term $\varepsilon_{\text{rank}}(c)$ in Eq. (11). This clarification makes it evident that the two terms partition the total gap without double-counting.

Foreshadow discussion about how a single design choice might affect multiple components.

This is a great suggestion. We have now added a short “cross-term interactions” paragraph at the end of §3.3 that makes this explicit.

A single design choice can shrink multiple error terms. When the confidence score is the maximum softmax probability (MSP), a better approximation of the true conditional $\eta$ not only lowers the approximation term $\varepsilon_{\text{approx}}(c)$ but also tends to align MSP more closely with $\eta_h$, thereby indirectly reducing the ranking error $\varepsilon_{\text{rank}}(c)$. Conversely, a non-monotone calibration head can slash $\varepsilon_{\text{rank}}(c)$ without improving $\varepsilon_{\text{approx}}(c)$.

We believe this forward-reference clarifies how model-capacity upgrades, richer pre-training, or loss-function tweaks can yield compound benefits, and we direct readers to the limitations section for a fuller discussion of such interactions.

---

4) Numerous issues with calibration discussion.

Incorrect calculations in Appendix B.2.

We thank the reviewer for catching the errors in Appendix B.2. We have thoroughly revised this section to correct both the analytic expression for temperature-scaled confidence scores and the illustrative example. The previous equation for $s_T(x)$ incorrectly assumed a binary logistic form; we now provide the correct expression for arbitrary $K$:

$$s_T(x) = \frac{1}{1 + \sum_{j \ne j_\star} \exp[(z_j - z_{j_\star})/T]} = \frac{1}{1 + \sum_{j \ne j_\star} r_j^{1/T}}.$$

We also replaced the earlier example, which wrongly claimed a ranking swap, with corrected logits that do exhibit a swap at higher temperature: $s_1(x_1) = 0.576 > 0.512 = s_1(x_2)$ at $T = 1$, but $s_3(x_1) = 0.411 < 0.428 = s_3(x_2)$ at $T = 3$. Finally, we clarified the condition for reordering: that the sums over non-max logits, $S_1(T) = S_2(T)$, cross for some $T$.

Existing research [1,2] demonstrates that TEMP has a large effect on SC.

We thank the reviewer for pointing to prior work suggesting that temperature scaling (TS) improves selective prediction. The discrepancy arises from differences in evaluation setup. Our analysis (§3.4, App. B.2) focuses on ranking-based selective classifiers, where prediction is accepted based on the top-$c$ fraction ranked by $s(x)$. In this setting, monotonic transformations like TS preserve the ordering and thus do not affect the selective classification gap $\Delta(c)$ or selective accuracy at fixed coverage. In contrast, [1] uses a fixed score threshold $\tau$ (e.g., “accept if $s(x) > \tau$”), where TS can shift scores so that more correct predictions exceed $\tau$, improving metrics like AUROC. However, AUROC mixes overall accuracy and ranking quality, whereas our decomposition isolates the selective component by conditioning on rejection. As shown in §4.2, while TS may improve calibration, it does not affect $\Delta(c)$ unless it changes the ranking. We have clarified this in the revised draft.

Poor discussion on scoring/binning.

We appreciate the reviewer’s observation and agree that our original discussion of histogram binning was too brief. While histogram binning fits a monotone mapping, it does induce score ties by mapping wide score ranges to the same value; we now clarify in §3.4 and Appendix B.2 how such ties can—and in rare cases do—affect the selection ordering. We have also added reference to the mentioned works [2,3] on post-training logit normalization and label smoothing.

---

5) Discussion on future directions, including (i) connections to OOD; and (ii) uncertainty in open-ended generative (sequence) models.

We appreciate the reviewer’s pointer to several fast-moving research threads that lie just beyond the scope of our current study. While we think that these directions should be more thoroughly investigated in future work, we want to acknowledge these connections. Hence, we have created a new Appendix F (“Beyond the Closed-World Setting”) that provides two self-contained subsections (§F.1 and §F.2) outlining how our decomposition might be generalized and which technical obstacles remain. We also include the following summary and pointer in our conclusion section:

Our finite-sample gap decomposition lays the groundwork for a more unified reliability framework; extending it to (i) settings where out-of-distribution inputs must be rejected and (ii) open-ended language generation—where correctness is semantic rather than categorical—constitutes a promising agenda for future work that we outline in Appendix F.

We are happy to provide a more detailed overview of these sections upon the reviewer’s request.

---

6) Advice given in Appendix C.

We have clarified in Appendix C which checklist items derive from our theoretical and empirical results versus established best practices, and we’ve supported each recommendation with appropriate citations (including the reviewer’s suggestions). To avoid confusion, we now explicitly note [3] cautioning against the use of label smoothing (which notably was not part of our original recommendations). For techniques with mixed effects—such as binning—we cross‐reference tie‐breaking remedies and advise tuning bin granularity to prevent coverage cliffs.

---

7) Minor issues: Definition vs takeaways color; Sec. 2 accuracy-coverage tradeoff reference; MSP and SR are both used.

We thank the reviewer for bringing these issues to our attention. We have adjusted the colors to be more distinctive between definitions and take-aways, have added a reference to [9], and have replaced the single mention of SR in §2 with MSP.

---

We hope that our rebuttal has addressed the reviewer’s questions and concerns and that the reviewer considers raising their individual and overall score(s). Our work has already significantly improved thanks to their feedback. We are happy to further engage with them during the discussion period!

I thank the authors for their clarifications, as well as for their incorporation of my feedback. I apologise for the delayed reply, I have been travelling over the weekend. There are still a few aspects of their response that I am unsure about however.

1. Temperature scaling does not change the ranking of logits for a given sample over classes, keeping accuracy@100% coverage constant for a given model, and therefore the number of correct and incorrect samples constant. Galil et al [1] measure the change in AUROC before and after temperature scaling for each classifier, and demonstrate meaningful changes. As the number of in/correct samples doesn't change, AUROC is a proxy for ranking/binary classification performance, and will only change if the confidence ranking between samples changes. Besides, Cattelan and Silva [2] demonstrate TS has a large effect on their metric, normalized AURC for various individual classifiers. This is empirical evidence over hundreds of models that temperature scaling changes ranking.
2. I firmly believe that histogram binning should not be recommended if the objective is to improve selective classification. Typically the order of 10 bins is used -- quantising each bin effectively randomises all confidence scores within each bin, reducing intra bin discrimination to random. Table 4 in [10]'s appendix demonstrates how histogram binning always degrades selective classification.
3. I am wary about Appendix C, could the authors provide concretely which advice they have given in the revised version and the supporting references? After all [1,2] have demonstrated that optimising for top 1 accuracy may severely negatively impact ranking performance -- many of their worst-performing models use the choices suggested in the Shrink Approximation Error section. Moreover, [10] demonstrate in their Table 4 that vector/Dirichlet scaling does not give consistent improvements and often hurts ranking performance.

[10] Le Coz et al. Confidence Calibration of Classifiers with Many Classes NeurIPS 2024

I will first briefly respond to the authors' previous comment. I will then follow up in a later comment a summary of my thoughts on the paper.

1. I agree with the point the authors make about the fundamental potential impact of "corrective" vs "generative" approaches to the problem -- what I was more concerned about was the diminishment of the practical relevance of post-hoc approaches given their demonstrated usefulness in the literature in certain cases. I would be satisfied if the authors re-adjusted their wording to more clearly highlight how and when post-hoc approaches can be useful.
2. I was primarily concerned with firm recommendations in the checklist without sufficient clarification/verification on SC-based effectiveness. I had missed the authors' reply to reviewer 9vHG about tagging bullets (as well as other improvements to the checklist), which I believe alleviates this issue sufficiently.
3. I agree with the authors' that [12] do not formally disentangle the $\varepsilon$ terms presented in this work, and admit my use such terms in the previous comment was imprecise. However, the results in [2] still question whether SAT leads to improvements solely due to better ranking (as was the narrative presented in the original SAT paper), which call into question the placement of SAT in the checklist. With regards to Appendix E, although I unfortunately do not have the bandwidth to read the mathematics in detail, I am sceptical of the utility of "Loss Prediction advantage", since it is perfectly possible for a model with terribly calibrated probability scores to have zero $\varepsilon_\text{rank}$ (e.g. $p=0.51$ for all correct, $p=0.49$ for all errors). Besides, it seems like the definition of $h(x)$ is unclear in this section between outputing probability scores or hard 1-hot predictions.

We sincerely thank the reviewer for their in-depth feedback. We are especially grateful for their recognition of the paper’s structured presentation, clear theoretical framing, and its potential as a starting point for newcomers to SC. At the same time, we acknowledge the reviewer’s concerns regarding the novelty, formulation, and scope of evaluation. We have taken these suggestions seriously and provide clarifications, additional related work discussion, and rewordings throughout our rebuttal to address them in detail.

---

1.1) Purpose of lines 35-36.

We would appreciate additional clarification, as we are unsure what the reviewer is specifically asking.

---

1.2) Algebraic decomposition of Equation 27.

We apologize for the confusion—Eq. 27 is not intended as the tautological inequality $a \le a + b$. Rather, by adding and subtracting $\mathbb{E}[\eta_h \mid A_c^{\star}]$, we obtain an exact identity

$$\Delta(c) = \varepsilon_{\text{rank}}(c) + \varepsilon_{\text{approx}}(c) + \varepsilon_{\text{Bayes}}(c),$$

where each non-negative term isolates a distinct failure mode:
(i) $\varepsilon_{\text{Bayes}}$ is the irreducible label noise among the accepted points;
(ii) $\varepsilon_{\text{approx}}$ captures errors that persist even if ranking were perfect (e.g., due to limited model capacity or training); and
(iii) $\varepsilon_{\text{rank}}$ quantifies the loss due solely to mis-ordering examples.

This factorization is therefore diagnostic, not merely algebraic: by measuring the three components in Sec. 4, we show, for instance, that ranking error dominates on CIFAR-100, pointing practitioners toward improving uncertainty estimation rather than enlarging the model. We have added a concise explanation of this objective immediately before Eq. 27 to prevent further misunderstanding.

---

1.3) Relationship to e-AURC [D].

The reviewer is correct that our formulation of the SC gap shares similarities with the definition of the Excess-AURC (E-AURC) introduced in [D], as both quantify deviations from an ideal abstention policy. However, the two differ in both granularity and perspective. Our gap measures pointwise accuracy shortfall at a fixed coverage level relative to a perfect oracle with the same base accuracy, whereas E-AURC aggregates excess risk across all rejection levels relative to an optimal ranking function. To clarify this connection, we have explicitly acknowledged the similarity between our gap and E-AURC directly following Definition 4 in the updated draft. We would also be happy to include a more detailed comparison of the two performance metrics in the appendix.

---

2.1) Discussion of recent works [A,B,C].

We thank the reviewer for pointing us to these influential recent works. We have incorporated their insights in the revised manuscript as follows:

In §2 “Accuracy–coverage trade-off evaluation” we added:

Recently, [B] introduced the Area Under the Goals-Reweighted Curve (AUGRC), which multiplies accuracy by coverage before integration, thereby down-weighting the very-low-coverage tail that can dominate the traditional AURC.

Complementary to metric design, [C] provides a comprehensive benchmark that disentangles sources of uncertainty, underscoring the need for task-aware evaluation of selective classifiers.

In §4.2 “Setup” we added:

Our inclusion on the maximum-softmax-probability baseline is motivated by the large-scale study of [A], who find that MSP proves to be a hard-to-beat baseline.

---

2.2) [A] goes against our reasoning.

Certainly—here is a more concise version that integrates your collaborator’s feedback while keeping the overall length close to the original:

---

First, we note that [A] did not evaluate Self-Adaptive Training (SAT) or Deep Ensembles (DE). Instantiated to support our theoretical formulation, our empirical results in §4.2 and Appendix E.5 show that SAT and DE consistently narrow the selective-classification gap better than MSP—by improving ranking quality—and are also more reliable at predicting their own loss. Second, we agree that practitioners ultimately care about the best possible performance at the chosen coverage. In that sense, minimizing the selective-classification gap complements accuracy maximization: it reveals whether remaining error stems from noise, ranking, or capacity limits, and thus which lever to pull (e.g., stronger backbone vs. better uncertainty scoring). This disentanglement also clarifies whether improvements from compute-heavy methods like DE come from gap reductions or generic accuracy boosts—helping practitioners identify more efficient alternatives. Please let us know if we have misinterpreted your concern.

---

3.1) Role of isolated gap quantities is hardly new to anyone in the field.

We agree that practitioners often recognize each factor qualitatively. However, to our knowledge, no prior work (i) formalizes all five influences in a single finite-sample, coverage-uniform inequality, (ii) proves that every SC gap can be exactly upper-bounded by these terms, and (iii) demonstrates—via controlled experiments—that each term can be independently measured and targeted. Our decomposition thus turns familiar intuitions into a quantitative error budget that guides design: it shows why monotone post-hoc calibration cannot improve ranking, when added capacity yields diminishing returns due to Bayes noise, and how shift-induced slack can limit SC gains. If the reviewer knows of prior work with this combination of decomposition, bounds, and empirical disentanglement, we would be grateful for the reference and acknowledge it. See extended discussion for Reviewer VZwS.

---

3.2) More citations in App. C and in the main body.

This is a good suggestion; we agree with the reviewer that the checklist would benefit from additional citations to prior work that has proposed or validated the listed solutions. In the revised version, we will enrich Appendix C by explicitly citing the relevant methods alongside each entry, including both original sources and follow-up studies that confirm their utility in selective classification. At the same time, we will also include relevant citations in the main body. We hope that this will help readers trace the origins of each recommendation and better understand the contexts in which they have been shown to be effective.

---

3.3) Ranking insight and connection to AUROC, AUPRC.

We appreciate the reviewer’s observation that metrics like AUROC/AUPRC improve with better score ordering. Our goal is not to reiterate this general fact, but to make it operational in the context of selective classification. Specifically, we isolate the ranking error term and show that (i) monotone calibration methods cannot reduce it, while (ii) reranking techniques can. Unlike threshold-averaged metrics, SC directly evaluates accuracy at fixed coverage levels, where a poor ranking can have a more immediate cost. By embedding ranking error alongside Bayes noise, approximation error, and slack in a single inequality—and validating each term through controlled experiments—we provide practitioners with a diagnostic toolkit to identify what limits performance and how to improve it.

---

4.1) “Large scale” evaluation appears to be overclaiming.

Our paper devotes substantial space to (i) developing a theoretical decomposition of the selective-classification gap, (ii) identifying which calibration mechanisms can or cannot tighten it, and (iii) connecting selective prediction to loss prediction. The experiments are designed as controlled diagnostics that vary one interpretable factor at a time to isolate its effect on each bound term. That said, we agree that the phrase “large-scale evaluation” overstates the current scope. We have revised the paper to the more accurate wording “systematic, controlled evaluation on synthetic and vision benchmarks.” We also note our response to Reviewer VZwS, who requested additional LLM results.

---

4.2) Only four different SC methods are implemented: 1) MSP 2) TEMP 3) SAT 4) DE.

We chose these four methods not for sheer breadth but as controlled, representative instantiations of the key scoring strategies in our decomposition: MSP/TEMP to illustrate that monotone recalibration only affects approximation error, SAT as a feature‐aware reranker, and DE as an ensemble‐based reranker. This ablation‐style setup lets us cleanly attribute gap reductions to each mechanism. Although the implementations are simple, these methods are widely used because of their ease of implementation and the insights into how each method maps to our theoretical error terms are novel. We will clarify this in §4.2 and, if desired, include a brief appendix study of additional scoring approaches.

---

4.3) Many recommendations from App. C are not explicitly experimented on.

Indeed, the recommendations in Appendix C consist of a combination of insights from our work as well as best practices in the SC literature. In the revised Appendix C we now distinctly label each recommendation as either (i) an original insight from our theoretical insights and controlled experiments; or (ii) a best practice drawn from the SC literature. As previously mentioned, we have added targeted citations so that readers can immediately see the evidence supporting each guideline.

---

5) Enumerations in 4.1 & 4.2.

We have fixed this as per the reviewer's suggestion.

---

We hope that our rebuttal has addressed the reviewer’s questions and concerns and that the reviewer considers raising their individual and overall score(s). Our work has already significantly improved thanks to their feedback. We are happy to further engage with them during the discussion period!

I thank the authors for their detailed rebuttal.

While I share Reviewer 7Yt4’s view on the potential value of this work and would like to raise my score, I remain unconvinced about the additional contribution over the E-AURC. This is primarily due to the continued lack of clarity regarding the mathematical decomposition of the gap (which I have now outlined more explicitly) as well as the strong similarity to E-AURC, which has not been sufficiently addressed.

---

Points 1.1 and 1.2 should be seen combined. I am sorry if this led to confusion.

As the stated purpose is to explicitly reduce the SC gap, it is important to assess how the proposed decomposition helps to do this. As of now, I am unsure how it does so.

Based on the definitions in the manuscript (see 23 for $\varepsilon_{\text{rank}}(c)$, below 27 and Point 1 for $\mathbb{E}[\eta_h \mid A_c^\star]=\overline{acc}(a_{\text{full}}, c)$, below equation 26 step 1 for $acc_{c}(h, g)=\mathbb{E}[\eta_h \mid A_c]$) the following statement holds true:

$$\Delta(c) := \overline{acc}(a_{\text{full}}, c) - acc_{c}(h, g) = \mathbb{E}[\eta_h \mid A_c^\star] - \mathbb{E}[\eta_h \mid A_c] = \varepsilon_\text{rank}(c)$$

Therefore, based on this, the authors prove that (see answer 1.2 and proof in Theorem 1):

$$\Delta(c) = \varepsilon_\text{rank}(c) = \varepsilon_{\text{rank}}(c) + \varepsilon_{\text{approx}}(c) + \varepsilon_{\text{Bayes}}(c)$$

Therefore $\varepsilon_{\text{approx}}(c) + \varepsilon_{\text{Bayes}}(c)=0$ where each of these must satsify $\geq 0$. If this is not equal but greater than or equals, then it boils down to:

$$\Delta(c) = \varepsilon_\text{rank}(c) \leq \varepsilon_{\text{rank}}(c) + \varepsilon_{\text{approx}}(c) + \varepsilon_{\text{Bayes}}(c)$$

But the question is then: how does reducing the other error sources influence the ranking error based on this decomposition?

Based on this, it appears that the gap only measures the difference from the perfect ranking.

---

Point 1.3 -- Please analyze the similarity of the SC gap to the E-AURC based on Equations instead of a text discussion

I asked for an analysis of the difference between the proposed SC gap and the E-AURC, especially in the form of a mathematical formulation. Please do this here, and also add it to the paper.

To my understanding, the main argument is that the SC gap operates based on a single coverage point, whereas the E-AURC is only defined as an integral over the entire curve. However, the work introducing the E-AURC already shows that the excess risk can be computed for each coverage point (see Geifman et al. 2019 Figure 1).

Further, in the manuscript in Table 1, the measurement is called GAP when according to lines 227-229 the integral is taken (in the new results table it is also called Gap Area) which would makes the metric actually used for numerical comparisons identical to the E-AURC, if I am not mistaken based on the point above (1.1-2).

I would highly encourage renaming the measurement to the E-AURC in this case to give proper attribution to the people who originally proposed this measure.

It could be that this stems from my lack of knowledge. However, as the authors are definitely aware of this work (as they cite it in their manuscript) and are also aware of the similarity it is important that the differentiating factor between these two metrics is as clearly defined as possible.

This is paramount to ensure that future works can properly reference both works for their contributions.

---

4.3 I agree with Reviewer 7Yt4, please provide explicit examples, which advice is given alongside the citations.

I thank the authors for their detailed response.

---

I am fully satisfied with their Mathematical analysis: SC Gap vs. E-AURC as well as their proposed changes.

---

Regarding point 4.3.

Thank you for this detailed answer.

I am highly positive about these changes.

However, I would encourage the authors to also explicitly search for works that show that the stated changes improve Selective Classification Performance to give proper attribution to works that already showcased this explicitly.

Further, if you can not find explicit references or it does not directly follow from the experiments in this manuscript, add a differentiation to show that these are based on a theoretical analysis and remain to be shown explicitly.

I will leave it up to the authors to follow up on these changes.

I deem this important to ensure that practitioners reading this chapter can clearly differentiate between hypotheses/theoretical results and best practices, which have already been shown (alongside how, based on the citation) and are gathered in one place.

---

Regarding Point 1.1 and 1.2

I appreciate the clarification from the reviewers and would encourage them to write this exact reasoning down in equation 27.

However, I wish to clarify that I do not assume this statement to be true:

$$\overline{acc}(a_{\text{full}}, c) = \mathbb{E}[\eta_h \mid A_c^\star]$$

Rather, it is directly written in the manuscript as I already pointed out in my previous answer (directly below equation 27 in point 1 following "Explanation of the two labelled inequalities." ... "(rank) uses the definition of the oracle: $\overline{acc}(a_{\text{full}}, c) = \mathbb{E}[\eta_h \mid A_c^\star]$").

Am I not reading it correctly? Is this a mistake in the manuscript, or is this something else?

We thank the reviewer for their thoughtful feedback and kind words about the paper’s clarity, presentation, and theoretical soundness. We appreciate their suggestions regarding experimental scope, perceived novelty, and practical utility of the decomposition. Below, we clarify our contributions and address these concerns—most notably by adding results on LLM-based models and elaborating on how the gap informs practice.

---

1) Novelty and relevance of the presented decomposition.

We thank the reviewer for their kind words regarding our exposition. We respectfully contend, however, that our work offers both novel theoretical insight and a practical diagnostic that was not available before. We address each of your concerns in turn.

• Coverage-uniform finite-sample bound. To our knowledge, no previous result provides a decomposition that simultaneously (i) holds for every coverage level $c$, (ii) is stated at finite sample size n, and (iii) partitions the gap into four interpretable sources. Existing selective-risk PAC bounds fix $c$ a priori and collapse ranking and approximation into a single term.
• Separation of ranking error. By isolating $\varepsilon_\text{rank}$​ we show that any monotone post-hoc calibration (temperature scaling, isotonic, histogram) cannot shrink the gap (§4.4). This clarifies empirical disagreement reported in recent studies.
• Interpretable formalization. Through formalizing the gap decomposition in Equations (14) and (15) we provide the first error budget for abstaining systems. Researchers can now ask which piece limits current methods and design targeted algorithms, just as bias-variance decompositions shaped classical supervised learning.
• Controlled experiments for each bound quantity. We provide a set of controlled experiments that (i) verifies our theoretical insights; and (ii) allows us to independently measure and target each of the contributing bound components.

We hope this concise clarification demonstrates the novelty and practical value of our contribution and addresses your concern about significance. If the reviewer is aware of prior work that offers the same combination of formal decomposition, tight upper bounds, and empirical disentanglement, we would be grateful for the references and will gladly acknowledge and cite them in our revised version.

---

2) Can you expand the experiments to include newer models and datasets, notably with LLMs?

We agree that large language models (LLMs) are an important test-bed for our framework. Achieving the same level of controlled experimentation as in our synthetic and vision experiments, however, is inherently harder for three main reasons:

1. Uncertainty for generative models remains ill-defined. For classification-style prompts but even more so for generative tasks in particular, the community has not fully converged on how to translate sequence-level probabilities into abstention scores.
2. Prompting artefacts add variance. Small changes in in-context examples or decoding settings can swamp the effects we wish to isolate.
3. Training pipelines are opaque and inconsistent. Many LLMs are trained with undisclosed data mixtures, objectives, or fine-tuning procedures—introducing uncertainty about what distributions the model has implicitly seen and how it generalizes.

Despite these challenges, we have added a focused set of LLM experiments—summarized below and plotted in the revised manuscript—to demonstrate that our decomposition still diagnoses the gap.

1. Approximation error — scaling from 4B → 12B

We evaluate Gemma 3-IT 4B and 12B [1] on ARC-Challenge [2] (ARC-C, 25-shot) and MMLU [3] (5-shot, top-1) using the standard MSP score on the first answer token (no further fine-tuning).

Observation. Increasing capacity reduces the gap, mirroring the vision results on model expressiveness.

2. Bayes error — separating easy vs. noisy questions (4B)

Following the MMLU-Pro protocol [4], we partition the validation set into the easiest 25 % and noisiest 25 % questions (based on human–LLM agreement).

Observation. When intrinsic Bayes noise is low (easy questions), the gap nearly vanishes; when noise is high, the gap widens.

3. Ranking quality — calibration vs. re-ranking (4B)

We keep the backbone fixed (Gemma 3-IT 4B) and compare:

• MSP (raw logits)
• Temp. scaling (scalar $T$ fitted on a held out validation split)
• Deep ensemble of five LoRA-fine-tuned replicas

Observation. Temperature scaling lowers ECE yet leaves the gap untouched; the ensemble both calibrates and improves ranking, shrinking the gap—recapitulating our CIFAR-100 findings.

Overall, these results confirm that our decomposition extends to LLMs: capacity, Bayes noise, and ranking quality each contribute measurable terms. A full generative-text study (e.g. free-form question answering or code synthesis) will require new abstention semantics and is left for future work.

[1]: Team, Gemma, et al. "Gemma 3 technical report." arXiv preprint arXiv:2503.19786 (2025).

[2]: Clark, Peter, et al. "Think you have solved question answering? try arc, the ai2 reasoning challenge." arXiv preprint arXiv:1803.05457 (2018).

[3]: Hendrycks, Dan, et al. "Measuring massive multitask language understanding." ICLR 2021.

[4]: Wang, Yubo, et al. "Mmlu-pro: A more robust and challenging multi-task language understanding benchmark." NeurIPS 2024.

---

3) How does this analysis and how does the gap Delta(c) relate to a gap we can compute in practice based on the oracle selective classification rule? Or is there other ways to compute this gap?

Our analysis is built around the empirical gap that practitioners already compute: for any held-out dataset with ground-truth labels, we obtain (i) the model’s empirical accuracy–coverage curve by thresholding its confidence scores, and (ii) the oracle curve by retroactively ranking the same examples in descending order of realized correctness (i.e., 1 for correct, 0 for incorrect)—exactly the “oracle selective-classification rule” used in prior work. The coverage-uniform gap $\widehat{\Delta}(c)$ we decompose is simply the point-wise difference between these two curves, or, when desired, its integral over $c$. Thus every term in our bound upper-bounds a quantity that is directly observable from data; no additional assumptions or simulations are required. Alternative oracle choices (e.g. Bayes posterior ordering when noise estimates are available) plug into the same recipe and our decomposition still applies verbatim—only the definition of the benchmark curve changes. In short, the gap we analyze is exactly the gap researchers and practitioners already plot, and our results explain why that observable gap arises and which interventions (capacity, non-monotone calibration, shift-robustness, etc.) can provably shrink it.

---

4) Please make Figure 1 bigger so that its easier to read

We have fixed this in our updated paper draft.

---

We hope that our rebuttal has addressed the reviewer’s questions and concerns and that the reviewer considers raising their individual and overall score(s). Our work has already significantly improved thanks to their feedback. We are happy to further engage with them during the discussion period!

Dear reviewer,

Please respond to the author rebuttal as soon as possible - but at Aug 8, anytime on earth, at the latest. Please do not only submit the "Mandatory Acknowledgement", but also please respond with text to the author rebuttal. For example, explain why the author rebuttal convinced you, or why not.

Many thanks, Your AC.

Dear reviewer,

Today is the last chance to respond to the author rebuttal. Please do so today by end of day (anywhere on earth). Please do not only submit the "Mandatory Acknowledgement", but also please respond with text to the author rebuttal. For example, explain why the author rebuttal convinced you, or why not.

Many thanks, Your AC.

We thank the reviewer for their thoughtful and constructive positive review. We are glad the reviewer finds the paper is well-written, the analysis original, and the experiments carefully designed. We appreciate the concerns raised regarding the practical design takeaways and the role of overfitting, and we address both points in detail below.

---

1) It is hard to see practical, non-obvious, design guidelines for building selective classifiers coming out of this formulation?

We want to clarify that our contribution goes beyond establishing yet another bound: by decomposing the gap into five interpretable terms—Bayes noise, approximation, ranking, statistical, and misspecification slack—we expose concrete levers that practitioners can manipulate. In particular, the new ranking term clarifies why monotone calibration alone cannot close the gap: it leaves the score ordering intact, so any benefit is limited to full-coverage accuracy rather than selective performance. Conversely, non-monotone or feature-aware calibrators and model-ensemble methods do tighten the ranking term, while increasing capacity or distilling from larger teachers targets approximation error, and robust-loss or repeated-label strategies reduce Bayes noise. We distilled these insights into a step-by-step practitioner checklist in Appendix C, which, to our knowledge, is the first set of explicit, empirically validated design guidelines for approaching oracle-level selective classification—transforming our theoretical formulation into a practical toolkit.

---

2) What is the role of overfitting on the SC gap?

Our results are reported exclusively on unseen test data, so the kind of overfitting to the training set that inflates in-sample accuracy is automatically folded into the finite-sample decomposition: if a larger model memorizes training idiosyncrasies, its test-time full-coverage accuracy plateaus while its confidence scores sharpen, thereby enlarging the ranking term $\varepsilon_{\text{rank}}(c)$ and, when optimization becomes unstable, the miscellaneous slack $\varepsilon_{\text{misc}}(c)$. Overfitting to a narrow training distribution manifests similarly under shift, and we isolate that effect in §4.3 where $\varepsilon_{\text{misc}}$ grows with corruption severity. Overall, we observe that capacity improves the gap only when it also preserves good ranking and generalization—otherwise the decomposition correctly attributes the performance loss to ranking or optimization slack. We are happy to include this discussion in the paper.

---

3) Where are the promised concrete design guidelines for selective classifiers?

We include these concrete guidelines in Appendix C.

---

We hope that our rebuttal has addressed the reviewer’s questions and concerns and are happy to further engage with them during the discussion period!

Dear reviewer,

Now is time for the discussion with the authors. Can you please reply to their rebuttal, indicating if your issues were clarified and or answered? If that is not the case, please indicate why the answer was not satisfying.

Many thanks! Your AC

Dear reviewer,

Please respond to the author rebuttal as soon as possible - but at Aug 8, anytime on earth, at the latest. Please do not only submit the "Mandatory Acknowledgement", but also please respond with text to the author rebuttal. For example, explain why the author rebuttal convinced you, or why not.

Many thanks, Your AC.

Dear reviewer,

Today is the last chance to respond to the author rebuttal. Please do so today by end of day (anywhere on earth). Please do not only submit the "Mandatory Acknowledgement", but also please respond with text to the author rebuttal. For example, explain why the author rebuttal convinced you, or why not.

Many thanks, Your AC.