Conformal Linguistic Calibration: Trading-off between Factuality and Specificity

Thank you for finding our partial abstention motivation compelling and appreciating our conformal guarantee formalization. We address your concerns about component robustness and computational costs below.

Q1: Clustering Component Evaluation

While comprehensive evaluation is challenging without ground truth labels, we optimized each pipeline component through careful design choices:

Identifying Clusters: We initially tested entailment-based approaches but found NLI models overly sensitive to surface forms and noisy (McCoy et al., 2019). Embedding-based approaches work poorly for numeric cases (Naik et al., 2019; Tang et al., 2025). For example, in one of our test cases “2023” is more similar to “2026” than “2024” in embedding space. Since we can’t predict if an LLM output will be numeric, number-specific methods are also hard to apply.

Estimating Answer Multiplicity (How semantic equivalence is determined): We use embedding-based similarity (answer will be assigned to the most similar cluster, which we deem to be semantically equivalent) for answer multiplicity estimation (Section 3.1). Here using embedding-based similarity to identify semantic equivalence is more reliable as unique clusters have been identified and usually there will be one cluster that is very similar to the answer at hand. In this case, we only need to find the most similar cluster, not discover new ones. Previous research suggests that despite noisy clustering, uncertainty estimation from traditional clustering approaches remains reliable (Kuhn et al., 2023; Grewal et al., 2024). While clustering accuracy is hard to evaluate directly without ground truth label, it is indirectly evaluated together with our belief probing verification study described below.

Belief Probing: One of the authors conducted manual verification on over 50 examples from different prompt categories to see whether a unique answer is entailed by the summary. The manual verification study shows precision: 0.81, recall: 0.91 across different question types, indicating robustness.

We also would like to note that we use LTT for calibration, which is based on hypothesis testing and only requires an independent and identically distributed calibration set. Even if no pipeline components are perfect, the LTT guarantee remains valid, although the performance of each component does affect the optimality of the trade-off curve.

Q2: Computational Cost

We acknowledge the computational overhead of curating high-quality answer sets. To address this, our experiments demonstrate that hedging behavior can be distilled into base models for direct generation. Other relatively expensive approaches that require sampling also resort to similar ideas (Kuhn et al., 2023; Farquhar et al., 2024; Band et al., 2024).

Q3: Open Source and Technical Issues

We will fix all formatting issues, including the last line of Figure 2, and will open source our complete dataset and step-wise creation code with the camera-ready version.

References

• McCoy, R. Thomas, Ellie Pavlick, and Tal Linzen. "Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference." arXiv preprint arXiv:1902.01007 (2019).
• Naik, Aakanksha, et al. "Exploring numeracy in word embeddings." Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.
• Tang, Eric, Bangding Yang, and Xingyou Song. "Understanding LLM Embeddings for Regression." Transactions on Machine Learning Research.
• Kuhn, Lorenz, Yarin Gal, and Sebastian Farquhar. "Kuhn, Lorenz, Yarin Gal, and Sebastian Farquhar. "Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation." The Eleventh International Conference on Learning Representations.
• Grewal, Yashvir S., Edwin V. Bonilla, and Thang D. Bui. "Improving uncertainty quantification in large language models via semantic embeddings." arXiv preprint arXiv:2410.22685 (2024).
• Farquhar, Sebastian, et al. "Detecting hallucinations in large language models using semantic entropy." Nature 630.8017 (2024): 625-630.
• Band, Neil, et al. "Linguistic Calibration of Long-Form Generations." International Conference on Machine Learning. PMLR, 2024.

Thank you for your encouraging review and recognition of our paper's clarity and structure. We appreciate your thoughtful questions about implementation details.

Q1: CLC Generation and Verbalized Confidence

Thank you for providing this interesting perspective. Through manual examination, we observe that our generated responses typically avoid explicit confidence statements. This is intended, as our formulation aims to represent uncertainty through hedging rather than verbalized confidence reports. In our implementation, our few-shot prompts in belief probing maintain structural similarity to original claims while incorporating appropriate hedging. We agree with the reviewer that it will be really interesting to see how LLM can simultaneously utilize different uncertainty communication protocols to improve user experience. We will open-source all data and code to facilitate such investigations.

Q2: Performance Scaling with K

Rerunning experiments with different K values requires expensive re-clustering and belief probing. However, we’d like to note that LTT remains valid with K=20. The effect of adding more number of samples is that it provides more fine-grained uncertainty level that the model can fall back to, as they produce more unique clusters:

To summarize: Higher K provides more nuanced uncertainty control, but even K=20 maintains sufficient cluster diversity for effective calibration.

Q3: Limitation Discussion and User Preferences

You raise an important point about user perception. Our current limitation discussion focuses primarily on pipeline errors and guarantee validity. We will expand this to include discussion of how AI confidence communication impacts human decision-making, acknowledging the challenges in conducting human-based factuality trade-off evaluations due to known biases (Li et al., 2025; Ding et al., 2025).

References

• Li, Jingshu, et al. "As Confidence Aligns: Understanding the Effect of AI Confidence on Human Self-confidence in Human-AI Decision Making." Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 2025.
• Ding, Yifan, et al. "Citations and trust in llm generated responses." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 22. 2025.

Thank you for your encouraging review and for speaking high of our work's originality and significance. We address your three specific questions (as well as related weaknesses) below.

Q1: Belief Probing and Nested Property

Effectiveness of the current probing setup: We thank the reviewer for raising the concern about potential errors in the pipeline and their implications. To address this, one of the authors conducted manual verification of whether a unique answer is entailed by the summary. The manual verification study shows precision: 0.81, recall: 0.91 across different question types, indicating robustness of our current belief probing component.

Nested Property Violation: As the reviewer correctly noted, relying on the LTT framework fully absorbs this risk—it doesn't require perfect nesting properties. Even if belief probing occasionally breaks nesting (producing irregular trade-offs), LTT's theoretical guarantees remain valid. Also empirically we observe more smooth and effective trade-off in Figure 3, and monotonic relationship between CPMI and multiplicity threshold levels, validating the effectiveness of our implementation.

Q2: Distilled Model Generalization

While providing explicit guarantees for OOD rewriting is challenging, our experiments suggest that the learned trade-off rewriting behavior has some generalizability. While our model is trained on QA data, the evaluation is conducted on Core (FActScore-like biographical claims) that is different from the QA data we trained, yet maintains effective factuality-specificity trade-offs. The distillation learns general hedging principles rather than domain-specific patterns. We’ll make this clearer in the revision. In theory, there are more recent conformal prediction methods that can deal with distribution shift explicitly, by modeling the shift directly (Prinster et al., 2024). So we can potentially leverage those work in the future. We do hope that future work will find better and more efficient rewriting that achieves a better tradeoff with higher correspondence.

Q3: Semantic Clustering vs. Confidence Scores

Using semantic similarity instead of confidence scores doesn't affect LTT guarantee validity. In fact any method that produces nested prediction sets is valid for conformal prediction, regardless of how those sets are constructed. As discussed by the reviewer, the major benefit of our approach is that it leads to more coherent nested sets, making belief probing easier and producing more interpretable claims.

References

• Prinster, Drew, et al. "Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them)." International Conference on Machine Learning. PMLR, 2024.

Thank you for recognizing the importance of uncertainty communication in LLMs and our novel integration of possible-world semantics with conformal prediction. We address your concerns about clarity and technical presentation below.

Q1: Clarified Explanation of Preliminaries

Definition of Sphere: In possible world semantics, spheres represent different levels of certainty or likelihood around what we consider "normal" or expected. Possible worlds are organized into concentric spheres by accessibility relationships. A claim is necessarily true within a sphere if true in all accessible worlds, and possibly true if true in some accessible worlds. We’ll add more description in the revision with the additional page.

Intuitive Explanation: We represent claims as sets of possible worlds where they remain true. By making claims less specific (hedging), we make them compatible with more possible alternatives, reducing factual error risk while maintaining informativeness. This formulation reveals connections between claim generation and set prediction, enabling conformal prediction for risk control. We will make sure that we add more substantial explanations and illustrations to explain this idea providing additional space.

Q2: Related Work and Baseline Selection

We have a related work section in Appendix A. Our approach differs fundamentally from standard abstention methods by enabling partial abstention at the claim level. While existing methods refuse to answer entirely, we can respond with partial knowledge when confidence is relatively low (see Figure 3 example).

For baseline, we compare against the closest approach (Mohri and Hashimoto, 2024) in Section 4.1. While as mentioned by the reviewer there are other works that study LLM abstention and control the risk with conformal techniques, they mostly share a very similar structure to the baseline we compared to on short response QA (Yadkori et al., 2024). And from a previous study the sample-based uncertainty we adopt for our baseline is generally considered the best non-conformity score for this purpose (Cole et al., 2023). Our result clearly shows that standard abstention methods struggle under challenging settings like SimpleQA, where few prompts can be answered correctly with confidence. Also, at near-full abstention, their accuracy becomes unstable due to insufficient remaining examples.

Q3: Procedure Diagram and Implementation Details

Figure 2 Clarification: This serves as our system diagram, with subtitles corresponding to Section 3 paragraphs. We will align subtitle naming with paragraph headers and add clearer indications of the diagram's purpose.

Belief Probing Robustness: In general LTT remains valid as long as we have an independent and identically distributed calibration set, even if our components are not always correct. On top of this, one of the authors conducted manual verification on over 50 examples from different prompt categories to see whether a unique answer is entailed by the summary. The manual verification study shows precision: 0.81, recall: 0.91 across different question types, indicating robustness.

Q4: Cluster Analysis with Varying K Values

While it will be interesting to know how performance scales K, rerunning experiments with different K values requires expensive re-clustering and belief probing. However, we’d like to note that LTT remains valid with K=20. In general, the effect of adding more number of samples is that it provides more fine-grained uncertainty level that the model can fall back to, as they produce more unique clusters:

To summarize: Higher K provides more nuanced uncertainty control, but even K=20 maintains sufficient cluster diversity for effective calibration.

References

• Mohri, Christopher, and Tatsunori Hashimoto. "Language models with conformal factuality guarantees." Proceedings of the 41st International Conference on Machine Learning. 2024.
• Yadkori, Yasin Abbasi, et al. "Mitigating llm hallucinations via conformal abstention." arXiv preprint arXiv:2405.01563 (2024).
• Cole, Jeremy, et al. "Selectively Answering Ambiguous Questions." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.