Enhancing Hallucination Detection with Noise Injection

Thank you for your comments and suggestions. We apologize for the typo -- Figure 7 should be Figure 2 instead. Please see our response below.

[Tradeoff of Complication and Performance] Thank you for highlighting the trade-offs involved. We agree that the method introduces additional considerations. As such, the decision to adopt this approach would indeed depend on the specific application and its requirements. We appreciate your suggestion and acknowledge that future work could further explore these aspects to refine the effort-benefit tradeoff.

Weaknesses

[Statistical Significance] Thank you for raising this concern. To assess the statistical significance of our results, we report the 95% confidence intervals in the table below. Specifically, we use a bootstrap method to estimate the intervals: we sample five generations per question from a broader pool of 20 generations with replacement and a bootstrap sample size of 25 for GSM8K, TriviaQA, and CSQA. For ProntoQA, we use a bootstrap sample size of 50 due to higher data variability.

We also conduct a t-test to determine whether the changes are statistically significant. All datasets pass the t-test (significance level: $\alpha = 0.05$), with the following results: GSM8K ($t_\text{crit} = 1.677, t_\text{score} = 20.669$), CSQA ($t_\text{crit} = 1.677, t_\text{score} = 3.553$), TriviaQA ($( t_\text{crit} = 1.677, t_\text{score} = 19.259$), and ProntoQA ($t_\text{crit} = 1.661, t_\text{score} = 2.041$).

Given the relatively small answer space, calculating entropy using five samples provides sufficient precision to highlight the differences introduced by noise injection. For predictive probability and normalized predictive entropy—metrics that evaluate a significantly larger space encompassing all possible reasoning and answer sequences—five generations sampled from our pool of 20 are less likely to yield reliable Monte Carlo estimates. This limitation could potentially be addressed in future studies by increasing the number of samples, expanding the generation pool, or focusing exclusively on the answer string. Nonetheless, under our current setup, our experiments demonstrate that noise injection does not degrade performance under these measures.

[Sensitivity on Noise Magnitude] Thank you for the suggestion. In response, we have conducted a sensitivity analysis of noise magnitude on both CSQA and TriviaQA datasets. While we observe that the optimal noise magnitude varies across datasets, the results indicate that noise injection over a broad range of magnitudes consistently improves performance. We will update the manuscript to include these experiments, providing results across multiple noise magnitudes to ensure a comprehensive evaluation.

Thank you for your feedback. Regarding the theoretical grounding, we have addressed this concern by elaborating on how adjusting parameters like $T$, complements noise injection. Specifically, temperature modifies the sampling distribution while preserving the token likelihood order (e.g., $\text{Pr}(\text{token}_A) > \text{Pr}(\text{token}_B)$), whereas noise injection can reverse this order, offering a complementary effect. We have revised lines 80–84 and 226–229 in the manuscript to incorporate this discussion.

As for the additional complexity, we have already clarified that it represents a trade-off. Its value depends on the application context, and practitioners may weigh this against their specific goals and constraints. Moreover, the proposed approach does not introduce additional inference delay, ensuring practical applicability in latency-sensitive scenarios.

Thank you for your comments and suggestions. We apologize for the typo, and Figure 7 should be Figure 2 instead. We cleaned up the typos in the updated draft. Please see our response below.

Weaknesses

[statistical significance and t-test] Thank you for raising this concern. To assess the statistical significance of our results, we report the 95% confidence intervals in the table below. Specifically, we use a bootstrap method to estimate the intervals: we sample five generations per question from a broader pool of 20 generations with replacement and a bootstrap sample size of 25 for GSM8K, TriviaQA, and CSQA. For ProntoQA, we use a bootstrap sample size of 50 due to higher data variability.

We also conduct a t-test to determine whether the changes are statistically significant. All datasets pass the t-test (significance level: $\alpha = 0.05$), with the following results: GSM8K ($t_\text{crit} = 1.677, t_\text{score} = 20.669$), CSQA ($t_\text{crit} = 1.677, t_\text{score} = 3.553$), TriviaQA ($( t_\text{crit} = 1.677, t_\text{score} = 19.259$), and ProntoQA ($t_\text{crit} = 1.661, t_\text{score} = 2.041$).

Given the relatively small answer space, calculating entropy using five samples provides sufficient precision to highlight the differences introduced by noise injection. For predictive probability and normalized predictive entropy—metrics that evaluate a significantly larger space encompassing all possible reasoning and answer sequences—five generations sampled from our pool of 20 are less likely to yield reliable Monte Carlo estimates. This limitation could potentially be addressed in future studies by increasing the number of samples, expanding the generation pool, or focusing exclusively on the answer string. Nonetheless, under our current setup, our experiments demonstrate that noise injection does not degrade performance under these measures.

[Mistral Experiments] Thank you for the suggestion. In response to your request, we conducted additional TriviaQA experiments on Mistral, using noise magnitudes of 0 and 0.02, consistent with our GSM8K experiments on Mistral. Without noise injection, hallucination detection AUROC is 66.42, and with noise injection, AUROC improves to 69.44. Due to time and computational constraints, we focus on this representative evaluation, which further demonstrate the method’s generality.

[Figure Illustration] Thank you for your suggestion. Figure 2 is based on only 5 generations, which limits the granularity of the entropy values. We are open to any further suggestions on enhancing visual understanding.

Questions:

[Explanation of Better Significance on Answer Entropy] We address the question in response to [Statistical Significance]

[Ablation study in Table 4 on additional datasets] Thank you for your suggestions. Below we ablate on temperature and noise magnitude with CSQA dataset. As in Table 4 in the paper, noise injection improves detection effectiveness compared to no noise. We will updated the manuscript to include the results.

[Other Perturbation Methods] Thank you for the suggestion. While input query perturbations are interesting, they are beyond the scope of this work. We focus on model perturbations and leave this direction for future exploration.

Thank you for your valuable questions and comments! Please see our response below.

Weaknesses

[Statistical Significance] Thank you for raising this concern. To assess the statistical significance of our results, we report the 95% confidence intervals in the table below. Specifically, we use a bootstrap method to estimate the intervals: we sample five generations per question from a broader pool of 20 generations with replacement and a bootstrap sample size of 25 for GSM8K, TriviaQA, and CSQA. For ProntoQA, we use a bootstrap sample size of 50 due to higher data variability.

We also conduct a t-test to determine whether the changes are statistically significant. All datasets pass the t-test (significance level: $\alpha = 0.05$), with the following results: GSM8K ($t_\text{crit} = 1.677, t_\text{score} = 20.669$), CSQA ($t_\text{crit} = 1.677, t_\text{score} = 3.553$), TriviaQA ($( t_\text{crit} = 1.677, t_\text{score} = 19.259$), and ProntoQA ($t_\text{crit} = 1.661, t_\text{score} = 2.041$).

Given the relatively small answer space, calculating entropy using five samples provides sufficient precision to highlight the differences introduced by noise injection. For predictive probability and normalized predictive entropy—metrics that evaluate a significantly larger space encompassing all possible reasoning and answer sequences—five generations sampled from our pool of 20 are less likely to yield reliable Monte Carlo estimates. This limitation could potentially be addressed in future studies by increasing the number of samples, expanding the generation pool, or focusing exclusively on the answer string. Nonetheless, under our current setup, our experiments demonstrate that noise injection does not degrade performance under these measures.

[Complementary Effect and Pearson Correlation] We agree that sampling at different temperatures may yield a similar Pearson correlation. However, such correlations still reflect complementary effects between different sampling temperatures. While combining different temperatures to leverage their complementary effect is not straightforward, our algorithm demonstrates how noise injection and temperature-based sampling can be effectively combined to leverage their complementary effects.

[Fair Comparison against Adjusting T, Top-P, Top-K] We agree that performance could be enhanced by optimizing hyperparameters such as $T$, $\text{top-P}$, and $\text{top-K}$. To ensure a fair comparison, we present results with different $T$ values while using the default values for$\text{top-P}$ and $\text{top-K}$ in Table 4. We observe that noise injection improves performance across all tested temperatures. Exhaustively testing all possible configurations is infeasible. However, theoretically, adjusting $T$, $\text{top-P}$, and $\text{top-K}$ would complement noise injection. Specifically, these parameters alter the sample distribution but preserve token likelihood ordering (e.g., if $\text{Pr}(token_A) > \text{Pr}(token_B)$, this order remains unchanged). In contrast, noise injection can reverse this order, offering a complementary effect.

[Theoretical Insight and Comparison with Increasing T] Thank you for the feedback. We do not claim that perturbations at the hidden layer are more effective than output layer sampling. Instead, we suggest that combining both can be more effective due to their complementary effects, as discussed theoretically in our response to [Adjusting T, Top-P, Top-K] . We also edit line 80-84, 226-229 to include the discussion. Empirically, we show in Table 4 that noise injection and temperature adjustments have distinct effects. Specifically, increasing $T$ from 0.5 to 0.8 or 1.0 reduces hallucination detection AUROC, but adding noise at $T = 0.5$ introduces a different source of randomness and improves performance (lines 425–428).

[Section 4: Statistical Significance] Thank you for raising this concern. To assess the statistical significance of our results, we report the 95% confidence intervals in the table below. Specifically, we use a bootstrap method to estimate the intervals: we sample five generations per question from a broader pool of 20 generations with replacement and a bootstrap sample size of 25 for GSM8K, TriviaQA, and CSQA. For ProntoQA, we use a bootstrap sample size of 50 due to higher data variability.

We also conduct a t-test to determine whether the changes are statistically significant. All datasets pass the t-test (significance level: $\alpha = 0.05$), with the following results: GSM8K ($t_\text{crit} = 1.677, t_\text{score} = 20.669$), CSQA ($t_\text{crit} = 1.677, t_\text{score} = 3.553$), TriviaQA ($( t_\text{crit} = 1.677, t_\text{score} = 19.259$), and ProntoQA ($t_\text{crit} = 1.661, t_\text{score} = 2.041$).

Given the relatively small answer space, calculating entropy using five samples provides sufficient precision to highlight the differences introduced by noise injection. For predictive probability and normalized predictive entropy—metrics that evaluate a significantly larger space encompassing all possible reasoning and answer sequences—five generations sampled from our pool of 20 are less likely to yield reliable Monte Carlo estimates. This limitation could potentially be addressed in future studies by increasing the number of samples, expanding the generation pool, or focusing exclusively on the answer string. Nonetheless, under our current setup, our experiments demonstrate that noise injection does not degrade performance under these measures.

Questions:

[Prompting and Answer Extraction] We prompt the model using In-Context Learning examples and extract the final answers based on the formatting provided in these examples. We provide further details in Appendix B.1.

[Greedy Decoding Accuracy] With greedy decoding GSM8K reports 29.11% accuracy; CSQA reports 62.62% accuracy; TriviaQA reports 73.57% accuracy; ProtoQA reports 76.2% accuracy.

[Difference on Table 2 and Table 3] The results differ because Table 2 presents GSM8K performance under a single random seed, which is the same seed used in Figures 2 and 3 for the corresponding experiments. In contrast, Table 3 reports the average performance across 20 random seeds.

Thank you for your comprehensive review and constructive comments. We apologize for the typos and Figure 7 should instead be Figure 2. We have fixed the typos in the updated PDF. Please see our response below.

Weaknesses

[Related work] Thank you for your suggestion. Our work builds upon prior studies linking model uncertainty to hallucination (lines 31-35; lines 495-500) by introducing a new source of randomness. Since we modify intermediate layers, we also review relevant work on hallucination detection using intermediate layer representations (lines 501-508). While we focus on areas most relevant to our contributions, we welcome any specific references to further enrich our discussion.

[Single Model] Kindly note that we experimented with multiple models: LLAMA2-13B-Chat for our case study and alternative models -- LLAMA2-13B-Chat and Mistral -- in Table 6.

[Intro: Hallucination Definition] Thank you for raising this concern. In our experimental setup, model responses are coherent (i.e., not repetitive or unreadable) and their correctness cannot be determined without reference (see Line 3 in Table 1). In this context, incorrect answers are considered plausible, which aligns with our definition of hallucinations. We have updated line 161-170 to clarify this.

[Intro: Empirical Validation] Thank you for raising this concern. We empirically validate our hypothesis that hallucination cases are less robust to noise injection in Figure 2(a). Specifically, the hallucination cases (grey) exhibit higher entropy, indicating greater variance in responses with injected noise. We have updated line 89-90 to clarify this point.

[Section 2: Hidden State Perturbation] Thank you for raising this concern. To clarify, when h_l is perturbed, we first compute h_{l+1} from the perturbed h_l. If layer l + 1 is selected, we then apply noise to h_{l+1}. Thus, the noise is not only added to the residual stream. We have updated line 143-144, 268-269, 298-299 to better explain this process.

[Section 3: Significance and dataset size] Table 2, as a case study example, is over one single seed. For the same setup, we demonstrate the standard deviation with 20 random seeds Figure 4. We evaluate on GSM8K test set containing 1319 questions. We have updated line 160-161 to report the dataset size.

[Section 3: Noise Effect on Accuracy.] Thank you for raising this concern. Kindly note that we do not claim that within a single generation, the number of hallucination cases appear less with noise injection. Instead, we argue that the incorrect answers generated during hallucination are less consistent across generations with noise injected, making incorrect answers less likely to be selected by majority vote. As a result, this shift improves the likelihood of correct answers being chosen, thereby enhancing accuracy under the majority vote scheme. We update line 338-344 in the manuscript to clarify the explanation.

[Section 3: Separate section for GSM8K] Thank you for your feedback. Section 3 serves as our methodology section, detailing our approach through a case study on GSM8K. Section 4, by contrast, presents experimental results across multiple datasets. This distinction aims to improve clarity, but we are open to suggestions for reducing potential repetition.

Thank you for your comments and suggestions. Please see our response to your concerns below.

[Fixed Distribution] While we exemplify our method with a uniform distribution, the mean and variance do vary as we select different noise magnitudes. As clarified in lines 464–466 of Section 4.5, the sampling distribution depends on the specific LLM, enabling adaptation and supporting generalizability.

[Statistical Significance] Thank you for raising this concern. To assess the statistical significance of our results, we report the 95% confidence intervals in the table below. Specifically, we use a bootstrap method to estimate the intervals: we sample five generations per question from a broader pool of 20 generations with replacement and a bootstrap sample size of 25 for GSM8K, TriviaQA, and CSQA. For ProntoQA, we use a bootstrap sample size of 50 due to higher data variability.

We also conduct a t-test to determine whether the changes are statistically significant. All datasets pass the t-test (significance level: $\alpha = 0.05$), with the following results: GSM8K ($t_\text{crit} = 1.677, t_\text{score} = 20.669$), CSQA ($t_\text{crit} = 1.677, t_\text{score} = 3.553$), TriviaQA ($t_\text{crit} = 1.677, t_\text{score} = 19.259$), and ProntoQA ($t_\text{crit} = 1.661, t_\text{score} = 2.041$).

Given the relatively small answer space, calculating entropy using five samples provides sufficient precision to highlight the differences introduced by noise injection. For predictive probability and normalized predictive entropy—metrics that evaluate a significantly larger space encompassing all possible reasoning and answer sequences—five generations sampled from our pool of 20 are less likely to yield reliable Monte Carlo estimates. This limitation could potentially be addressed in future studies by increasing the number of samples, expanding the generation pool, or focusing exclusively on the answer string. Nonetheless, under our current setup, our experiments demonstrate that noise injection does not degrade performance under these measures.

Dear Reviewers,

The authors have provided responses - do have a look and engage with them in a discussion to clarify any remaining issues as the discussion period is coming to a close in less than a day (2nd Dec AoE for reviewer responses).

Thanks for your service to ICLR 2025.

Best, AC