FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guarantees

Thank you for your detailed reviews and questions you raise to help improve our paper!

W1: statistical significance testing

We've conducted bootstrap analysis for 95% confidence intervals. Results available in: https://anonymous.4open.science/r/ICML_rebuttal-8905/icml2025__FactTest-2.pdf. Confidence intervals for all datasets will be included in our revision.

W2: Temperature could affect UQ

Temperature affects uncertainty quantification but doesn't limit FactTest, which controls type I error for any score regardless of temperature. Experiments with different temperatures (see link above) show that while accuracy varies, type I error remains below α.

W3: Score function selection impact

For any score function $\hat\eta$, the type I error is always controlled below $\alpha$ (Theorem 2.1). Type II error is nearly optimal with an inflation depending on $\epsilon_\eta=\inf_{H\text{ increasing}}\|H\circ\hat\eta-\eta\|_\infty$, which is the deviation between the oracle score $\eta$ and the used score $\hat\eta$ up to increasing transformation $H$ (Theorem 2.2).

W4: Multiple generations and latency

FactTest works with any score function, including single-generation ones. FactTest-cls in Table 14 (Section D.6) directly predicts answer correctness without multiple generations, with negligible runtime overhead. We expect more efficient score functions developed in the future to further enhance FactTest.

W5: sensitive to threshold choices

I am afraid there is a misunderstanding of our type I and II error guarantee. Our method selects threshold that guarantees type I error control below α with probability ≥1-δ for any user-specified α,δ. Moreover, our power analysis indicates that the threshold selected always possesses nearly optimal type II error as long as the score function captures the correctness of generated answers. Therefore, the performance of the selected threshold is always guaranteed.

W6: False negatives

FNs occur when score functions poorly separate correct/incorrect samples. This requires better score functions, orthogonal to our contribution of providing statistical guarantees for any score function.

W7:binary approach may oversimplify factuality

We currently use binary accept/reject decisions at significance level α. A natural extension is to output the answer together with the largest confidence level $1-\alpha$ for which the answer is rejected, providing a spectrum of factuality..

W8: when/why fails

While FactTest controls type I error for any score function, poor score functions increase type II error. With a constant function, FactTest would reject all answers to maintain type I error bounds.

Q1: long-form

Extending to long-form is our future direction. We see two approaches: 1.Document-level analysis: We could apply our framework with score functions designed to measure overall factuality, treating the entire response as a single unit. 2.Claim-level analysis: Formulate as a multiple testing problem and extends FactTest from controlling false positive rate to false discovery rate. For an answer with $m$ claims $c_1,...,c_m$, we will have: $H_{0,j}$: Claim $c_j$ is not correct. $H_{1,j}$: Claim $c_j$ is correct.

Q2: runtime overhead

Inference Runtime of FactTest shown in https://anonymous.4open.science/r/ICML_rebuttal-8905/icml2025__FactTest-2.pdf Single-generation function (e.g., FactTest-cls) add negligible overhead.

Q3: Open-domain settings with limited calibration data

If the correct answers for in-distribution questions $P_{q,M(q),a}$ are limited, one possiblility is to incoporate OOD questions $\tilde P_{q,M(q),a}$ for which correct answers are available, and then apply our method in section 3 to address the distribution shift in the calibration samples. Our method in section 3 only requires the oracle rule (optimal score function), of judging whether an answer is correct or not for a question, remains the same, $P(y=1|q,M(q))=\tilde P(y=1|q,M(q))$. Then it remains to estimate the density ratio of incorrect answers $\frac{dP_{q,M(q)|y=0}}{d\tilde P_{q,M(q)|y=0}}$, which equals to

$ \frac{dP_{q,M(q)|y=0}}{d\tilde P_{q,M(q)|y=0}}=\frac{dP_{q,M(q),y=0}\tilde P_{y}(0)}{d\tilde P_{q,M(q),y=0}P_y(0)}=\frac{dP_{q,M(q)}\tilde P_y(0)}{d\tilde P_{q,M(q)}P_y(0)}\propto\frac{dP_{q,M(q)}}{d\tilde P_{q,M(q)}}. $

Since the multiplicative constant $\frac{\tilde P_y(0)}{P_y(0)}$ in the density ratio only affect the efficiency of rejection sampling, provided the range $B$ of uniform random variables is large enough compared to $\frac{\tilde P_y(0)}{P_y(0)}$, we can estimate the density ratio $\frac{dP_{q,M(q)}}{d\tilde P_{q,M(q)}}$ based on merely unlabeled question-generated answer pairs $(q,M(q))$, which doesn't rely on the correct answers at all. Therefore, we believe our method is still applicable even if correct answers for in-distribution questions are limited.

Thank you for your feedback and suggestions. We are glad that you acknowledge our work’s motivation, method and theoretical claims. Here we provide responses and additional experimental results to address your concerns.

W1:guarantees are dependent on training data pairs

In Section 3, we specifically address the covariate shift setting, which allows the distribution of testing question-answer pairs to differ from the distribution of calibration pairs, provided that the oracle rule (optimal score function) for judging whether an answer is correct remains consistent. We have already incorporated experiments on OOD datasets in Section 5.4, where Figure 4 demonstrates that FactTest-O (our extension for out-of-distribution domains) effectively controls Type I error and significantly outperforms baseline methods in terms of accuracy. These results empirically validate our theoretical extension to covariate shifts, showing that our framework maintains its statistical guarantees even when applied to question distributions different from those in the calibration set.

W2:More baselines

Thank you for suggesting additional related works. We believe there may be a misunderstanding about the nature of our contribution. FactTest is fundamentally a meta-framework that works with any score function (including uncertainty quantification methods), deriving a statistically guaranteed threshold to determine whether to reject an answer, rather than proposing a new uncertainty scoring mechanism itself.

Regarding the specific papers mentioned:

[1] proposes a metric to evaluate self-consistency that could serve as a score function within our FactTest framework rather than as a competing baseline.

[2] is a benchmarking paper that does not propose a new UQ or hallucination detection method, but rather evaluates existing approaches with new metrics.

[3] proposes UQ methods for black-box hallucination detection which could serve as score functions for FactTest. We have implemented their UDEG approach as a score function within our framework (FactTest-udeg), with results shown below.

[4] is already implemented in FactTest as score function, which we denote as FactTest-se in our experiments.

[5] trains a classifier to predict statement truthfulness. We have implemented this as SAPLMA and compared it with FACTTEST. Additionally, we show how it can be used as a score function within our framework (FactTest-saplma).

Here we provide experiments of FactTest compared with [5]. We also implement [5] and udeg in [3] as a score function within FactTest to further demonstrate how these UQ methods could work within our framework.

These results further demonstrate how existing uncertainty quantification methods can be integrated into our framework, with FactTest providing statistical guarantees on Type I error control while maintaining or improving accuracy.

Thank you for your constructive feedbacks. We are glad that you acknowledge the validity of our framework and experimental design. Here we provide responses to your concerns one by one.

W1:Threshold of SCGPT

We acknowledge that using a threshold of 0.5 for SelfCheckGPT may not be optimal. In our framework, SelfCheckGPT is more suitable as a score function where thresholds are derived with explicit control over Type I error, rather than as a direct baseline. FactTest aims to determine thresholds that guarantee control over Type I error for any score function. In fact, Table 13 in Section D.6 demonstrates the integration of SelfCheckGPT as a score function in our framework. For a more appropriate comparison, we have implemented SAPLMA [1] as a baseline, a classifier specifically designed for hallucination detection that predicts whether an answer is correct. The accuracy (%) and Type I error (numbers in parentheses) performance are:

[1] Azaria, Amos, and Tom Mitchell. "The internal state of an LLM knows when it's lying."

W2:why accuracy, why not AUC-PR, AUCROC, or AURCC

The primary aim of FactTest is rigorous control over Type I error by explicitly setting a rejection threshold for any score function (including UQ) with statistical guarantees. When determining whether an answer is correct using a given score function, a threshold must be selected to distinguish correct from incorrect responses, with each threshold corresponding to a specific Type I error rate. Rather than reducing Type I error, FactTest determines the threshold for any score function that can statistically ensure the Type I error below user-specified $\alpha$. This operating point is crucial for high-stakes applications where accepting a hallucinated answer even rarely can be unacceptable.

• Why not AUC-PR, AUCROC, or AURCC: These metrics evaluate the overall effectiveness of uncertainty scores across all possible thresholds. They do not capture performance at the fixed operating point (a specific threshold at a user-specified $\alpha$) required by our method. These metrics would be more appropriate for evaluating the underlying score functions rather than the thresholding mechanism itself. Users could use such metrics to compare different score functions before applying our FactTest framework.
• Why accuracy: Since we determine the threshold at a user-specified α and use it to reject answers considered incorrect, we naturally report Type I error (to verify our theoretical guarantees) and accuracy on willingly answered questions (to demonstrate practical utility). This approach directly measures the performance at our chosen operating point rather than averaging across all possible thresholds.

W3:type 1 and type 2 don't compare to any baselines

Our framework aims to control Type I error under a user-specified level α, meaning the threshold determined by FactTest depends on α. UQ methods typically output scores without rejection thresholds, while baselines like R-Tuning use fixed rejection rules, resulting in only one Type I error rate. Our Type I error figures (Figure 1) demonstrate performance with different α values, showing that Type I error can almost always be controlled below α using FactTest. Adding a baseline like R-Tuning would simply show a horizontal line, as it doesn't offer the flexibility of varying threshold levels that FactTest provides.

W4:more recent papers on uncertainty

We appreciate this feedback and will incorporate discussion of more recent works in our next version to ensure comprehensive coverage of related methods, including but not limited to:

"[1] proposes a computationally efficient method that leverages semantic diversity to detect hallucinations. [2] revisits standard uncertainty metrics and highlights the limitations of naive entropy-based methods. [3] provides an effective and computationally efficient method to quantify predictive uncertainty in LLMs."

[1] Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs

[2] Rethinking Uncertainty Estimation in Natural Language Generation

[3] Improving Uncertainty Estimation through Semantically Diverse Language Generation