Calibrating Expressions of Certainty

We appreciate the reviewer’s thoughtful feedback and provide clarifications below:

Comparison to Baseline Calibration Methods: We apologize that our description of the experimental results in Table 1 did not make it sufficiently clear that our goal is not to significantly improve calibration of numerical scores (which is what Table 1 evaluates) but rather show that even if one reduces the output natural language expressions of certainty to scalar values (by simply taking the mean of the confidence distribution), our method remains competitive and does not compromise performance in terms of accuracy. We emphasize that our method's key advantage isn't in outperforming baseline calibration methods, but in directly operating on and producing natural language certainty phrases - a property that existing methods fundamentally lack. This makes our approach uniquely suited for improving human calibration in real-world settings, as it provides actionable guidance (e.g., suggesting radiologists use "May" instead of "Present" in their reporting to mitigate overconfidence) rather than abstract probability adjustments that can't be easily understood by humans. The goal is to maintain strong calibration performance while enabling calibration in natural language settings where traditional calibration methods cannot be meaningfully applied.

Outlier Expression and Mapping to the Middle: The observed mapping of certainty phrases to "middle" categories (e.g., "Maybe") in Figure 3 reflects dataset-specific characteristics, e.g., radiologists have high false negative and false positive rates, rather than a limitation of our calibration method. For instance, Appendix C.2 demonstrates a different calibration map that shifts the use of certainty phrases from high confidence to low confidence to reduce overconfidence, e.g., "Definitely" to "Possibly", ..., "Unlikely" to "Impossible".

Dense Math Derivation: To ensure clarity, it’s necessary that we provide concrete mathematical derivations of our method. If the reviewer has specific questions about the derivation, we will gladly clarify and expand to improve readability.

Clarifying Examples & Deeper Analysis: To address this, we have now included qualitative examples in Appendix B.2, illustrating how radiologists and language models use certainty phrases in the analyzed datasets. We will gladly provide additional analyses and examples if the reviewer has specific suggestions in mind that will probe the utility of the proposed approach.

Improve Calibration of Radiologists: Figure 3 demonstrates how radiologists' calibration improves by following our calibration method perfectly, with ECE reducing from 0.215 to 0.138 for atelectasis and from 0.276 to 0.177 for edema. We are conducting a follow-up clinical study, that is outside the scope of this paper, to examine how receptive radiologists are to calibration-improving suggestions provided by our method, e.g., use "May" or "Likely" instead "Present" in your reporting to address overconfidence.

We appreciate the reviewer’s thoughtful feedback and provide clarifications below:

Focus on Binary Classification: Our focus on binary classification aligns with the real-world problem of calibrating radiologists' binary decisions about the presence of pathologies in medical images. Similarly, calibrating a language model’s confidence in its free-form answers involves binary classification of its correctness. This foundational case supports a wide range of practical applications, e.g., free-form language model generation.

Complexity of Our Approach: To ensure clarity, it’s necessary that we provide concrete mathematical derivations of our method. To improve accessibility, we now supply code for estimating ECE, plotting reliability diagrams, and our proposed calibration method in supplementary material.

Scalability of Optimization: While natural language allows for arbitrary certainty phrases, in practice, commonly used subsets are small. Furthermore, optimal transport solvers like the Sinkhorn algorithm scale efficiently to high-dimensional problems, ensuring practical usability.

Calibrating Humans: Our study is the first to assess radiologists' calibration and provide tools to improve their calibration. We completely agree with the reviewer that improving human calibration is challenging and warrants further research. For instance, future work could explore how humans respond to different strategies for improving their calibration so that we can more effectively intervene. While we do not yet have definitive answers to the behavioral questions, we believe our work serves as an important first step toward addressing them.

Informativeness of Reliability Diagram: Figure 4 demonstrates that our approach produces smooth calibration curves that are easier to interpret. The deviation of the calibration curve (red) from the identity line, weighted by the score density (gray), provides an intuitive sense for the value of ECE. In contrast, binned reliability diagrams make it harder to visually assess calibration from calibration curves alone (see if you can guess which model is the strongest by just looking at calibration curves in Appendix C.3 figures). In addition, conclusions about model calibration from binned diagrams can vary significantly depending on the binning strategy used.

Interpretability of Calibration Method: One goal of our post-hoc calibration method is to improve radiologists' calibration. Figure 3 shows how radiologists could achieve better calibration by following our method perfectly. We are conducting a follow-up clinical study to evaluate the method’s effectiveness in the real world. Our method is interpretable because it translates directly into actionable suggestions for radiologists, e.g., use “May” instead of “Present” and “Likely” in your reporting to mitigate overconfidence. In contrast, methods like histogram binning or Platt scaling are less practical, as they do not provide clear guidance on how radiologists can adjust their language in reporting to address miscalibration.

Confidence Distributions for LLM Experiments: In Table 3 & Figure 5 (Figure 7 in updated draft), we found that predefining a fixed set of certainty phrases ($K=12$) for the model to choose from yields better calibration than alternatives. This fixed set of certainty phrases and their corresponding confidence distributions is generated by gpt-4o using 5th prompt in Table 5. These confidence distributions reflect the language model's interpretation of the associated phrases rather than being treated as definitive ground truth.

Certainty Phrases Preprocessing for Radiology Report Dataset: We used Llama-3 to act as a “smart regex” to extract radiologists’ own report-level certainty phrases. As an example, when encountering “The opacity at lower right corner is likely pneumonia and possibly edema” in a radiology report, Llama-3 is prompted to extract (“Likely”, “Pneumonia”) and (“Possibly”, “Edema”).

We appreciate the reviewer’s thoughtful feedback and provide clarifications below:

Motivation for Modeling Certainty Phrases as Distributions: Certainty phrases are central to our study as they reflect how radiologists or language models communicate confidence. Representing certainty phrases as distributions is a modeling choice that captures their inherent variability and subjectivity. More concretely, modeling certainty phrases as distributions aligns with how individuals naturally express uncertainty. People rarely associate phrases like "Highly Likely" with specific probability values (e.g., 89%), but rather with ranges of confidence. In addition, doing so enables us to distinguish between phrases like "Exactly Equal Probability" and "Maybe", which might both mean roughly 50% probability but differ in their certainty (narrow vs. wide density function). Using distributions to represent certainty phrases is further supported by clear evidence from survey of radiologists’ perception of different certainty phrases. Even if individuals are internally consistent, their interpretation of certainty phrases varies meaningfully across a population. For instance, "Possibly" was mapped to diverse probability ranges: 4.9% assigned it to (0-5%), 17.6% to (10-25%), 16.9% to (25-75%), and 0.7% to (75-90%), resulting in a population-level empirical distribution for “Possibly” that has non-uniform density and overlapping support with other empirical distributions (See Figure 1). This variation demonstrates why simple scalar values or binning are insufficient. Our framework naturally subsumes these simpler approaches as special cases (scalar values correspond to delta distributions, binning employs uniform distributions) while enabling us to model the rich structure we observe in how people (and thereby models) communicate uncertainty.

Clarify Limitations: Flexible certainty phrases: Our method is not limited to a fixed set of phrases; we focus on commonly used ones for clarity. New phrases can be incorporated if their meanings can be quantified, for example, through human surveys or prompting language models. For instance, Appendix B.3 (on-the-fly setup) demonstrates that our approach applies to LMs that can generate arbitrary Beta distributions to express their confidence rather than selecting from a fixed set of predefined phrases. Free-form text: Our approach handles free-form text by parsing (pathology, confidence) pairs from radiology reports, enabling calibration analysis for classifying specific pathologies. Calibration for arbitrary free-form text is not well-defined without framing it within some classification setup. For instance, in our LM experiments, we prompt models to generate free-form text and provide confidence ratings, enabling the analysis of calibration relative to model correctness in answering the question in context.

Comparison to Baseline Calibration Methods: We apologize that our description of the experimental results in Table 1 did not make it sufficiently clear that our goal is not to significantly improve calibration of numerical scores (which is what Table 1 evaluates) but rather show that even if one reduces the output natural language expressions of certainty to scalar values (by simply taking the mean of the confidence distribution), our method remains competitive and does not compromise performance in terms of accuracy. We emphasize that our method's key advantage isn't in outperforming baseline calibration methods, but in directly operating on and producing natural language certainty phrases - a property that existing methods fundamentally lack. This makes our approach uniquely suited for improving human calibration in real-world settings, as it provides actionable guidance (e.g., suggesting radiologists use "May" instead of "Present" in their reporting to mitigate overconfidence) rather than abstract probability adjustments that can't be easily understood by humans. The goal is to maintain strong calibration performance while enabling calibration in natural language settings where traditional calibration methods cannot be meaningfully applied.

Latex Margin Overflow: Thank you for catching it; we will correct it.

We appreciate the reviewer’s thoughtful feedback and provide clarifications below:

Curated Dataset Details: We have added additional details about our curated dataset in Appendix B.1.

Evaluation on Public Datasets such as UNLI:

We appreciate the reviewer bringing UNLI to our attention. While UNLI includes human-provided probability annotations, these are not necessary for evaluating a language model's calibration when it outputs certainty phrases about an answer's correctness—ground-truth labels suffice for such evaluations. Moreover, our existing evaluations on public datasets SciQ (where language models are well-calibrated) and TruthfulQA (where calibration is poor) already provide complementary insights into model calibration across diverse settings. As such, UNLI does not add to the diversity of setups we already investigate in the paper.

If the reviewer’s suggestion was to evaluate human annotators’ calibration, we note that the UNLI annotations are scalar probabilities rather than certainty phrases, which means they do not provide sufficient information to evaluate the utility of our method designed for calibrating certainty phrases. Consequently, we believe UNLI does not provide additional information beyond what is already captured by the datasets we have used. That said, we welcome the reviewer’s suggestions for additional datasets that could provide new insights into calibrating natural language expressions of (un)certainty.

Comparison to Baseline Calibration Methods: We apologize that our description of the experimental results in Table 1 did not make it sufficiently clear that our goal is not to significantly improve calibration of numerical scores (which is what Table 1 evaluates) but rather show that even if one reduces the output natural language expressions of certainty to scalar values (by simply taking the mean of the confidence distribution), our method remains competitive and does not compromise performance in terms of accuracy. We emphasize that our method's key advantage isn't in outperforming baseline calibration methods, but in directly operating on and producing natural language certainty phrases - a property that existing methods fundamentally lack. This makes our approach uniquely suited for improving human calibration in real-world settings, as it provides actionable guidance (e.g., suggesting radiologists use "May" instead of "Present" in their reporting to mitigate overconfidence) rather than abstract probability adjustments that can't be easily understood by humans. The goal is to maintain strong calibration performance while enabling calibration in natural language settings where traditional calibration methods cannot be meaningfully applied.

Probability Simplex Notation: Thank you for catching this typo; we will correct it.

Instruction Template & Reproducibility of LM Results: We clarify that we don’t use in-context learning but rather a zero-shot instruction template to elicit verbalized confidence by predicting beta distribution parameters directly (in Table 5). The code for replicating these experiments is now provided as supplementary material, and we encourage the reviewer to verify its reproducibility.

We sincerely appreciate the time and effort the reviewer and meta-reviewer have put into evaluating our work. We acknowledge that a slam-dumk experiment demonstrating the full potential of our approach is a lingering issue. As part of our ongoing work, we are actively applying this method to clinical problems/datasets, where we believe it can have significant impact. We hope this paper serves as an important first step in that direction and contributes to the broader discussion on LLM calibration. Thank you again for your thoughtful feedback!