HaDeMiF: Hallucination Detection and Mitigation in Large Language Models

Dear Reviewer goPV,

We sincerely appreciate your thoughtful review and valuable feedback. We have carefully considered all of your comments and suggestions, and have made corresponding revisions to the manuscript. Below, we provide our detailed responses to each of your comments.

Q1: The proposed framework introduces more tunable parameters than existing model calibration methods.

The volume of parameters introduced by our method is comparable to that of the LITCAB and P(IK) methods, while our approach consistently achieves superior performance. Moreover, while the label smoothing method requires fine-tuning the LLM, our method outperforms it even without the need for such fine-tuning.

Following your valuable comments, to ensure a more thorough comparison with fine-tuning-based methods, we further compare HADEMIF with two other approaches: Calibration-Tuning [1] and R-Tuning [2], which fine-tunes the LLMs for model calibration and hallucination mitigation. The results of these comparisons are presented in Tables 1 and 3 (highlighted in blue) of our revised manuscript. Although both Calibration-Tuning and R-Tuning fine-tune LLMs, their performance falls short of ours due to their inability to achieve fine-grained prediction calibration and their limitation to generating only binary confidence scores, respectively. All the above findings demonstrate the effectiveness of our approach in comparison to existing baselines.

Additionally, we conducted a comparative analysis between the performance of our approach and the plain fine-tuning of LLMs using LoRA. The results for NQ and TruthfulQA, presented below, demonstrate that our approach consistently outperforms the plain fine-tuning method.

Moreover, our experiments have demonstrated that simply increasing the number of tunable parameters does not always lead to performance improvement. As shown in Figs. 3(b) and (c) of the main text, optimal performance is achieved when the MLP network has 2 or 3 layers. However, increasing the number of layers to 4, while adding more tunable parameters, leads to performance degradation. Additionally, Fig. 3(d) indicates that increasing the maximum number of cut points in the D3T network does not necessarily enhance performance. These findings collectively suggest that the key to improving performance is not merely increasing the number of tunable parameters.

Q2: The comparison of the performance against fine-tuning with varying numbers of parameters.

Thank you for your constructive suggestion. We have evaluated the performance of HADEMIF against fine-tuning with different numbers of parameters in LLMs. Specifically, we fine-tuned the Llama2-7B model using LoRA with rank values set to 8, 16, and 32. The corresponding results are presented in the table below.

The results show that HADEMIF consistently outperforms other approaches across different numbers of tunable parameters, underscoring its effectiveness under varying training complexities. We also observe that using a low rank in LoRA can already achieve strong performance, consistent with the findings presented in the original LoRA paper. These experiments have been added to Appendix A.5 in our revised manuscript.

Moreover, from the results presented in Tables 6 and 7 of the manuscript, our method steadily demonstrates effectiveness across a wide range of model sizes (from 1.5B to 30B) and architectures, highlighting its broad effectiveness.

[1] Calibration-tuning: Teaching large language models to know what they don’t know.

[2] R-tuning: Instructing large language models to say ‘I don’t know’.

Dear Reviewer goPV:

We hope that our revisions and clarifications have adequately addressed your concerns. Your constructive feedback has been instrumental in enhancing the quality of our work. Should any aspects still require further elaboration, or if you have any additional questions, we would be pleased to address them. In light of the improvements and clarifications we have made, we would greatly appreciate it if you could consider adjusting your scores accordingly.

Thank you once again for your time and effort in reviewing our work.

Sincerely,

Authors

Dear Reviewer goPV:

We sincerely appreciate the time and effort you have invested in reviewing our work. As we approach the close of the author-reviewer discussion period in one week, we kindly inquire whether our clarifications and revisions have sufficiently addressed your concerns. Your feedback on the current revision is extremely valuable, and we would be truly grateful for any further suggestions you may have to provide us with an additional opportunity to refine our work.

Once again, we deeply appreciate your constructive comments and the contribution you have made to help strengthen our submission. Should any concerns remain, we would be more than happy to address them in full.

Best regards,

Authors

Dear Reviewer goPV, we sincerely thank you for your thoughtful and thorough review of our manuscript, which has significantly strengthened the quality of our manuscript. As the discussion stage is ending soon, we would like to kindly inquire if you have any remaining questions or concerns. Any feedback is greatly appreciated and will undoubtedly help further enhance our work. Moreover, we would be truly grateful if you would consider increasing the score in light of the revisions and clarifications we have made, which were directly informed by your insightful feedback. Thanks again for your efforts in reviewing our paper!

Q3: Comparison with more training-based approaches.

We completely agree that incorporating additional training-based approaches would enhance the fairness of our comparison. Our previous experiments have already included several such training-based methods. Specifically, the label smoothing method involves fine-tuning the LLMs using LoRA with the label smoothing loss. Additionally, the P(IK) and LITCAB methods necessitate the training of additional linear layers that are introduced. The rationale for comparing both training-based and non-training-based methods lies in previous studies, which suggest that, in the context of model calibration and hallucination, a training-based method is not always superior to a non-training method. Thus, we believe that incorporating comparisons across different types of methods is meaningful.

For a more thorough comparison, we further incorporated two other fine-tuning methods for LLMs: R-Tuning [1] and Calibration-Tuning [2], which fine-tunes the LLMs for hallucination mitigation and calibration, respectively. Since Calibration-Tuning produces a single score for the entire generation, it cannot evaluate calibration at the claim level, as each generation typically includes multiple claims. Therefore, we do not assess its performance on paragraph-level tasks, following LITCAB. The results of these additional comparisons are presented in Tables 1 and 3 (highlighted in blue) of our revised manuscript. Although both Calibration-Tuning and R-Tuning fine-tune LLMs, their performance falls short of ours due to their inability to achieve fine-grained prediction calibration and their limitation to generating only binary confidence scores, respectively.

Q4: Generalization across tasks.

As the two hallucination detection networks exhibit a certain degree of generalization across various tasks, applying our approach to a new task requires only fine-tuning the networks, rather than retraining them from scratch. To illustrate this, we conduct experiments on the NQ and TriviaQA datasets, using detection networks pre-trained on the SciQ dataset. We evaluate performance under three conditions: no fine-tuning, fine-tuning the two hallucination detection networks for 5 epochs, and fine-tuning for 15 epochs. As shown in the table below, only fine-tuning for 5 epochs can achieve reasonable performance, while fine-tuning for 15 epochs results in performance nearly identical to that obtained from training the model from scratch.

The need for additional training time is a common drawback of methods that introduce extra modules, such as adapters in the parameter-efficient fine-tuning of LLMs and the LITCAB method. However, the generalization ability of our approach across various tasks partially mitigates this limitation. Moreover, the generalization ability of the two detection networks highlights the potential for developing a universal hallucination detection network by training on large, diverse datasets. We intend to investigate this avenue in our future research.

Q5: Handling complex scenarios, such as open-ended creative writing tasks, where no clear factual answer exists.

Our approach introduces two hallucination detection networks to capture hallucination biases, such as overconfidence and inaccuracy, in both the output and internal spaces of LLMs. These networks then generate logit adjustment terms to calibrate the LLMs’ prediction distribution, thereby eliminating hallucinations and ensuring more reliable predictions.

In scenarios where true answers are unavailable, such as in open-ended creative writing tasks, self-supervised training objectives should be utilized, with the predicted probability distribution being calibrated using the outputs from the two hallucination detection networks, like Eq. (3) of the main text. As you mentioned, creative content may exhibit relatively high uncertainty and low consistency, which could lead to it being identified as hallucinations. To fully mitigate this issue, it may be necessary to incorporate additional strategies, such as human feedback or the integration of more creative data, during the training of the hallucination detection networks. We will explore the effectiveness of our approach on open-ended tasks in future research.

[1] R-tuning: Instructing large language models to say ‘I don’t know’.

[2] Calibration-tuning: Teaching large language models to know what they don’t know.

Dear Reviewer x16Z,

We sincerely appreciate your insightful review and valuable comments and suggestions. We have carefully considered each of them and made corresponding revisions in the updated version of our manuscript. Below, we provide our detailed responses to your comments:

Q1: Training data source.

We apologize for not clearly explaining the training data of the two hallucination detection networks in the main text, which may cause confusion for the readers. Our approach does not require additional annotations for hallucinations, meaning that no manual or automatic labeling process is involved. The two detection networks are optimized using the original training data from the dataset described in Appendix A.1 by minimizing the loss of the LLMs, with predictions calculated using Eq. (3) in the main text. During training, these two networks capture hallucination biases, such as overconfidence and inaccuracies, in both the output and internal spaces of the LLMs and generate adjustment terms to calibrate the LLMs’ prediction distribution, with the goal of maximizing the token probabilities for correct generations while minimizing the likelihood of incorrect ones.

We have clarified this point in Section 3.1.2 of the revised version of our manuscript (highlighted in blue). The added content is as follows: "Notably, our method does not require any additional hallucination annotations. It only requires training the two hallucination detection networks by minimizing the loss of LLMs on the original training set, with predictions computed as outlined in Eq. 3. This process aims to maximize the token probabilities for correct generations while reducing the likelihood of incorrect ones."

Q2: Interpretability of D3T.

We humbly respond to this comment from the following aspects:

• How D3T achieves its interpretability: The interpretability of D3T arises from its tree-based structure, which employs a soft binning function for feature splitting and utilizes the Kronecker product operation to determine the final node of the tree. This tree structure enables the revelation of decision rules for hallucination detection, which reflect how the combination of prediction characteristics influences hallucination detection. Although the importance of various input features is not explicitly incorporated during the training process of D3T, the method’s reliance on feature splitting enables the computation of their information gains, which can be used to assess their significance, thereby highlighting key characteristics for hallucination detection.
• Conducted analysis for its interpretability: In Appendix A.7, we visualized the first three layers of the decision tree in Fig. 5, where each path represents a hallucination detection rule, illustrating how the combination of various prediction characteristics determines the existence of hallucination. The findings reveal that hallucinated generations are generally associated with low consistency, high uncertainty, and small margins. Additionally, in Section 4.6, we assess the importance of various prediction characteristics (Fig. 3(a)) by calculating their information gains. The results indicate that consistency and uncertainty are the most significant factors, while the additional characteristics we introduced also contribute meaningfully to the detection process.
• Broader impact for future research: We believe that a thorough exploration of the derived decision rules could inform future research directions, such as refining directions for improving LLMs, enhancing user trust and acceptance, and enabling greater control over LLMs through real-time monitoring and analysis. These will be the focus of our future research.

Based on your valuable feedback, we have included additional explanations about the interpretability of D3T in Section 3.1.1, Section 4.6, and Appendix A.7 in our revised manuscript.

Dear Reviewer x16Z:

We hope that our clarifications and revisions have adequately addressed your concerns. We believe the paper has been significantly improved thanks to your comments. If any points remain unclear, or if you have additional questions or suggestions, please feel free to share them. Given the improvements and clarifications we have made, we would be most grateful if you could consider adjusting your scores accordingly.

Thank you once again for your time and effort in reviewing our work.

Sincerely,

Authors

Q2: The effectiveness of the combination of a series of prediction characteristics.

We humbly respond to this comment from the following aspects:

• The limitation of utilizing single metrics and the superiority of using a combination of characteristics: In the response to Question 1, our method consistently achieves higher AUROC and PCC values compared to those based on individual metrics, highlighting that reliance on a single characteristic is insufficient for achieving satisfactory detection performance. Moreover, the detection cases presented in Table 12 further underscore the limitations of using a single metric for hallucination detection. For instance, prior studies have often assumed that predictions with high uncertainty and low consistency are indicative of hallucinations. However, as shown in Table 12, even generations exhibiting high consistency and low uncertainty can still manifest hallucinations. Additionally, the results in Table 3 demonstrate that our approach consistently outperforms previous hallucination detection and mitigation baselines, validating the effectiveness of leveraging a combination of characteristics.
• The rationale behind characteristics selection: Although some of our adopted prediction characteristics have appeared in previous studies, to the best of our knowledge, no hallucination detection method has yet integrated these characteristics and used a network to automatically learn the function between these metrics and hallucinations. Based on our findings, this approach achieves superior performance and provides new insights for future hallucination research. To further clarify the rationale behind our selection of these characteristics, we have provided additional explanations in Section 3.1.1 of our revised manuscript. The revised content is as follows: "First, we consider the commonly used hallucination detection metrics from previous studies, specifically uncertainty and consistency. Next, we incorporate three additional metrics that reflect prediction confidence and are commonly employed in previous machine learning tasks, such as fairness evaluation and sample weighting (Zhang et al., 2020; Ross & Doll’ar, 2017; Jin et al., 2024)."
• Significance of our extracted characteristics: In Section 4.6, we have evaluated the significance of our extracted characteristics by calculating their respective information gains, as illustrated in Fig. 3(a). The results demonstrate that all metrics contribute to the hallucination detection process, with consistency and uncertainty proving to be particularly influential.

Q3: Ablation studies for the comparison between D3T and DNDT.

We fully agree with your suggestion to conduct an ablation study comparing D3T with DNDT. Unlike DNDT, which employs a fixed architecture resulting in numerous redundant branches, our D3T network features a flexible structure that dynamically adjusts the number of cut bins for each input feature during training. This adaptability allows for better alignment with the local characteristics of the data, thereby reducing the risk of overfitting. The effectiveness of both models as hallucination detection networks has been compared on the NQ and SciQ datasets. The results indicate that employing D3T as the hallucination detection network consistently outperforms the DNDT model. These experiments have been included in Appendix A.6 of the revised version of our manuscript.

Dear Reviewer QEQq,

We deeply appreciate your thorough review and valuable comments. We have carefully considered all your comments and have addressed them in the revised manuscript. Below are our detailed responses to each of your comments.

Q1: The gap between calibration and hallucination and the relationship between ECE and acc@50.

We respectfully address this concern from the following perspectives:

• The relationship between hallucination and calibration: Their relationship is mutually reinforcing. Calibrating LLMs plays a crucial role in detecting and mitigating hallucinations, while the mitigation of hallucinations, in turn, contributes to enhancing model calibration. In our approach, two lightweight networks are introduced to capture hallucinations in both the output and internal spaces of LLMs and generate adjustment terms to calibrate the model’s probability distribution, with the goal of maximizing the token probabilities for correct generations while minimizing the likelihood of incorrect ones, thereby achieving both hallucination mitigation and model calibration.
• Further clarification in our manuscript: We have incorporated the aforementioned content into Section 1 of our revised manuscript. Moreover, to further clarify the relationship between hallucination mitigation and model calibration, we have also expanded the related work section (Section 2) to include additional discussions on model calibration (highlighted in blue).
• Relationship between ECE and acc@50: Based on the performance improvements observed across all four metrics, our approach demonstrates effectiveness in both hallucination mitigation and model calibration. The two metrics you mentioned, ECE and acc@50, evaluate model performance from different perspectives. There is no strict proportional or inverse relationship between them, as indicated by their calculation manners.
• Additional experiments for hallucination detection: To provide a more comprehensive evaluation of the hallucination detection capability of our approach, we compared it against three additional detection baselines using the LLaMA-7B model. For our approach, the hallucination detection score is derived from the D3T model. The evaluation metrics include two widely adopted hallucination detection indicators: AUROC and PCC. As shown in the results below, our method consistently outperforms previous detection approaches that rely solely on a single characteristic. These experiments have been included in Appendix A.4 of our revised manuscript.

[1] Out-of-distribution detection and selective generation for conditional language models.

[2] Uncertainty estimation in autoregressive structured prediction.

[3] Towards collaborative neural-symbolic graph semantic parsing via uncertainty.

Q6: Comparison with finetune-based methods and methods designed for hallucination mitigation.

Our previous comparison has included some training-based methods. For example, the label smoothing approach that we have compared involves fine-tuning the LLMs using LoRA with the label smoothing loss (as detailed in lines 376-377 of the main text). Moreover, the P(IK) and LITCAB methods necessitate the training of additional linear layers that are introduced.

Following your valuable feedback, to provide a more comprehensive and balanced comparison, we further compared our approach with two other fine-tuning methods for LLMs: R-Tuning [1], which fine-tunes the model to mitigate hallucinations, and Calibration-Tuning [2], which fine-tunes the model to improve calibration. Since Calibration-Tuning produces only a single unified score for the entire generation, it cannot evaluate calibration at the claim level, as each generation typically contains multiple claims. Consequently, following LITCAB, we do not evaluate the performance of Calibration-Tuning for paragraph-level tasks. The results of these additional comparisons are presented in Tables 1 and 3 (highlighted in blue) of our revised manuscript. Although both Calibration-Tuning and R-Tuning fine-tune LLMs, their performance falls short of ours due to their inability to achieve fine-grained prediction calibration and their limitation to generating only binary confidence scores, respectively.

Q7: Computational costs.

Taking Llama2-7B as an example, compared to fine-tuning LLMs using LoRA (with a rank of 8), the time required to update the two hallucination detection networks per iteration is only 0.9% of the time needed to update the LLMs. Moreover, the time required for characteristics extraction per iteration is equivalent to the time needed to update the LLMs using LoRA. As observed, the majority of the time in our approach is spent on characteristics extraction, particularly for the calculation of the consistency metric. If the consistency metric is excluded, the time spent on feature extraction per iteration reduces to less than 0.1% of the time required for updating the LLMs using LoRA. Consequently, to improve efficiency, we plan to explore alternative metrics and more efficient methods for assessing model consistency in future work. We have added the above details regarding the computational overhead of our approach in Appendix A.12 of the revised manuscript.

[1] R-tuning: Instructing large language models to say ‘I don’t know’.

[2] Calibration-tuning: Teaching large language models to know what they don’t know.

Q4: Initialization of the weights of the two hallucination detection networks.

We apologize for the omission of the initialization settings for the two hallucination detection networks in the original manuscript. Corresponding contents have been added to Appendix A.2 of our revised manuscript (highlighted in blue). For the MLP network, we utilize He initialization [1], as this initialization approach is well-suited for layers with ReLU activation functions. For the D3T model, following the initialization approach in DNDT, all parameters are initialized using Xavier initialization with a uniform distribution [2].

To identify the optimal initialization settings for the two networks, we compared four different initialization configurations under the condition of the same number of training iterations (40 epochs). The MLP network considers two initialization methods: He initialization and Xavier initialization with a uniform distribution (denoted as X-u). Moreover, D3T considers two initialization methods: X-u and Xavier initialization with a normal distribution (denoted as X-n). The comparison results are shown in the table below:

Based on the above results, He initialization, specifically designed for ReLU activation functions, proves to be more advantageous for initializing the MLP network. For the D3T network, the performance difference between Xavier initialization with uniform and normal distributions is minimal. The above experiments have been added to Appendix A.9 of our revised manuscript.

Q5: The interpretability of metrics.

The interpretability of our approach arises from the use of an interpretable model (i.e., D3T) for the output-space hallucination detection network, which facilitates the revelation of the decision rules for hallucination detection associated with the prediction characteristics (as shown in Fig. 5 of the Appendix). Furthermore, the relative importance of various characteristics in hallucination detection can be analyzed by calculating their information gains thanks to its feature-splitting operation (as shown in Fig. 3(a) in the main text).

Based on our understanding, the interpretability of the metrics you referenced appears to be related to how the state of each characteristic correlates with the occurrence of hallucinations. According to our analysis, there is no definitive conclusion on this matter. For instance, previous studies that rely solely on a single metric often assume that high consistency or low uncertainty implies the absence of hallucinations. Yet, as demonstrated in Table 12 of our manuscript, this assumption does not always hold, as generations with high consistency or low uncertainty can still be hallucinations. Nevertheless, the decision rules obtained from the D3T model can reflect how the combination of various prediction characteristics determines the presence of hallucinations. Therefore, extensive analysis of these rules can uncover certain patterns. From our analytical experiments, non-hallucinated answers typically exhibit high consistency, low uncertainty, and large margins, although this relationship is not absolute. We are open to further clarification if our explanation differs from your intention.

[1] Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification.

[2] Understanding the difficulty of training deep feedforward neural networks.

Dear Reviewer QEQq:

We hope that our explanations and revisions have adequately addressed your concerns. All of your comments are valuable and have been taken into careful consideration and fully addressed in the revisions. If you find them satisfactory, we kindly ask that you take these improvements into account when reconsidering the score. We remain open to any further comments, follow-up questions, or suggestions, and would greatly appreciate your feedback.

Thank you once again for your time and effort in reviewing our work.

Sincerely,

Authors

Dear Reviewer QEQq:

We sincerely appreciate the time and effort you have invested in reviewing our work. As we approach the close of the author-reviewer discussion period in one week, we kindly ask whether you are satisfied with the clarifications and revisions we have made. Your feedback on the current revision is extremely valuable, and we would greatly appreciate any further suggestions you may have to provide us with an additional opportunity to refine our work.

Once again, we would like to express our sincere gratitude for your thoughtful review and the contributions you have made to strengthening our submission. Should any further concerns remain, we are fully committed to addressing them.

Best regards,

Authors

i’d like to thank the authors for the response. Your experiments have clarified most of my doubts. I have modified the score.

Dear Reviewer 76SS,

We express our sincere gratitude for your thorough review and insightful comments on our paper. We have carefully considered all your comments and addressed them in our revised manuscript. Below are our specific replies to your comments:

Q1: More explanations about DNDT.

We fully acknowledge your suggestion to provide a more detailed explanation of the DNDT model. In response, we expanded the description of DNDT in Section 3.1.1 of our revised manuscript (highlighted in blue). The revised contents are as follows: "D3T is an adaptation of the Deep Neural Decision Tree (DNDT) model (Yang et al., 2018), which is a tree model implemented using a neural network. This method introduces a soft binning function for feature splitting, which is achieved through a linear layer with Softmax as the activation function, and utilizes the Kronecker product operation to determine the final node of the tree."

Moreover, to systematically compare the performance of DNDT and D3T, we conducted ablation studies using both networks as hallucination detection networks for the output space. The experimental results, presented below, demonstrate that employing the D3T model as the hallucination detection network yields superior performance, owing to its flexible structure during training.

These experiments have been included in Appendix A.6 of our revised manuscript. Additionally, we have carefully reviewed and revised the entire manuscript to ensure that all descriptions are clear and accurate.

Q2: Limited explanation of the interpretability benefits of D3T in practical application.

We humbly respond to this comment from the following aspects:

• Challenges that can be solved: Existing studies employ different predictive characteristics for hallucination detection, yet the relative importance of various metrics remains unclear. Moreover, these methods often rely on manually defined functions or thresholds, lacking proper guidance.
• How D3T achieves its interpretability: The interpretability of D3T arises from its tree-based structure, which employs a soft binning function for feature splitting and utilizes the Kronecker product operation to determine the final node of the tree. This tree structure enables the revelation of decision rules for hallucination detection, which reflect how the combination of prediction characteristics influences hallucination detection. Although the importance of various input features is not explicitly incorporated during the training process of D3T, due to its reliance on feature splitting, the information gain for various characteristics can be computed to reflect their importance. Consequently, when visualizing the tree, the characteristics can be organized in layers according to their importance, in descending order.
• Conducted analyses using its interpretability: In Appendix A.7, we visualized the first three layers of the decision tree (Fig. 5), where each path represents a decision rule reflecting how the combination of prediction characteristics determines the existence of hallucinations. From the results, hallucinated generations are generally associated with low consistency, high uncertainty, and small margins. Additionally, in Section 4.6, we visualize the importance of various characteristics in Fig. 3(a). The results indicate that while uncertainty and consistency play a dominant role, other predictive characteristics, such as margin and predicted probability, are also useful for hallucination detection.
• Benefits and broader impact: The interpretability of D3T can facilitate a clearer understanding of the relationship between predictive characteristics and hallucination detection, as well as an analysis of which metrics are most critical for detecting hallucinations. We believe that a thorough exploration of the derived decision rules could offer valuable guidance for future research, such as refining the directions for improving LLMs and establishing stronger control over LLMs through real-time monitoring and analysis.

Building on the responses above, we have provided further explanations regarding the interpretability of D3T in Section 3.1.1, Section 4.6, and Appendix A.7 of the revised manuscript.

Q3: Clarification of the choice of metrics and their weights within D3T.

Previous studies typically utilize uncertainty and consistency for hallucination detection, generally assuming that hallucinations are associated with high uncertainty and low consistency. Our approach continues to leverage these two metrics without any assumptions. Additionally, to capture knowledge within the output space more comprehensively, we explore three other indicators: predicted probability distribution, predicted margin, and logits vector. Generally, a large maximum predicted probability, a wide margin, and a large norm of the logits vector indicate high confidence in the prediction. These metrics that reflect prediction confidence are commonly applied in previous machine learning tasks, such as fairness evaluation (as you referenced), sample weighting, and logit adjustment [1][2][3].

Following your helpful suggestion, we have added more details about the rationale behind the selection of prediction characteristics in Section 3.1.1 of our revised manuscript. The added content is as follows: "First, we consider the commonly used hallucination detection metrics from previous studies, specifically uncertainty and consistency. Next, we incorporate three additional metrics that reflect prediction confidence and are commonly employed in previous machine learning tasks, such as fairness evaluation and sample weighting (Zhang et al., 2020; Ross & Doll’ar, 2017; Jin et al., 2024)."

Based on the analyses presented in Section 4.6 of our manuscript, all extracted characteristics contribute effectively to hallucination detection. Their importance, ranked from highest to lowest, is as follows: consistency, uncertainty, margin, probability, and logits norm. Additionally, we do not need to manually assign weights to these characteristics; instead, we learn the function between prediction characteristics and hallucinations automatically using the proposed D3T network.

Q4: Model applicability with increasing training complexity, particularly in the context of LoRA.

To assess the model applicability with increasing training complexity, we analyze how the performance of our approach changes with an increase in the LoRA rank. The results on the NQ dataset, using HADEMIF with fine-tuning, are presented in the table below.

The results show that HADEMIF consistently outperforms other approaches across different rank values, highlighting its applicability. We also observe that using a low rank in LoRA can already achieve strong performance, consistent with the findings presented in the original LoRA paper. These experiments have been added to Appendix A.5 in our revised manuscript, which is highlighted in blue.

Moreover, from the results presented in Tables 6 and 7 of the manuscript, our method steadily demonstrates effectiveness across a wide range of model sizes (from 1.5B to 30B) and architectures, highlighting its broad effectiveness. To further validate the scalability and versatility of our approach, and to highlight its potential future directions for efficiency improvements, we also include a complexity analysis of the two hallucination detection networks and an evaluation of the computational overhead in Appendices A.11 and A.12 of our revised manuscript.

[1] Geometry-aware instance-reweighted adversarial training.

[2] Adaptive logit adjustment loss for long-tailed visual recognition.

[3] Focal loss for dense object detection.

Dear Reviewer 76SS:

We hope that our explanations and revisions have adequately addressed your concerns. We sincerely appreciate your thoughtful and insightful feedback, which has been invaluable in enhancing the quality of our paper. Should any points remain unclear, or if you have further questions or suggestions, please do not hesitate to share them. We would be deeply appreciative if you could take our clarifications and improvements into account when reassessing our work.

Thank you once again for your time and effort in reviewing our work.

Sincerely,

Authors

Dear Reviewer 76SS:

We sincerely appreciate the time and effort you have devoted to reviewing our manuscript. As the author-reviewer discussion period comes to a close in the coming week, we kindly inquire if you are satisfied with the clarifications and revisions we have made. Your feedback on the current revision would be extremely valuable, and we would be truly grateful for any further suggestions you may have to provide us with an additional opportunity to refine our work.

Once again, we express our heartfelt gratitude for your constructive comments and your contribution to strengthening our submission. Should any concerns remain, we would be more than happy to address them in full.

Best regards,

Authors

Dear Reviewer 76SS, we sincerely thank you for your thoughtful and thorough review of our manuscript, which has significantly strengthened the quality of our manuscript. As the discussion stage is ending soon, we would like to kindly inquire if you have any remaining questions or concerns. Any feedback is greatly appreciated and will undoubtedly help further enhance our work. Moreover, we would be truly grateful if you would consider increasing the score in light of the revisions and clarifications we have made, which were directly informed by your insightful feedback. Thanks again for your efforts in reviewing our paper!

Dear Reviewer ApSu,

We sincerely thank you for your thoughtful review and constructive comments. All the comments have been carefully considered and addressed in our revised manuscript. Below are our detailed responses to your comments:

Q1: How does the framework perform with larger models or in real-time applications?

Our framework is model-agnostic, enabling seamless adaptation to larger models. Moreover, only an additional parameter cost of less than 2% of the total parameters in the LLMs is introduced, ensuring scalability and efficiency.

Experiments have been conducted to evaluate the performance of our approach across models of varying sizes and architectures. As demonstrated in Tables 6 and 7 of the Appendix, our method consistently delivers strong performance across models ranging from 1.5B to 30B. In the context of real-time applications, LLMs fine-tuned using our approach can be directly deployed, avoiding an increase in inference time while simultaneously enhancing model calibration and mitigating hallucinations.

Q2: Can the framework be adapted to other types of hallucinations?

The proposed framework incorporates two hallucination detection networks to identify hallucination biases, such as overconfidence and inaccuracies, in both the output and hidden spaces of LLMs. These networks then generate logit adjustment terms that calibrate the predicted probability distribution of the LLMs, aiming to maximize the token probabilities for correct generations while reducing the likelihood of incorrect ones. This approach can mitigate various types of hallucinations, including factual, logical, and coherence-related ones, thereby improving the reliability of the model's generations.

To further evaluate the effectiveness of our approach in hallucination detection, we compared HADEMIF with three additional detection methods, in addition to those presented in our manuscript. The metrics include the area under the receiver operator characteristic curve (AUROC) and Pearson Correlation Coefficient (PCC), both of which are commonly used in previous studies. The comparison results are presented in the table below, where our approach demonstrates superior hallucination detection performance compared to previous studies that rely solely on a single aspect of the metric. These experiments have been included in Appendix A.4 of the revised version of our manuscript (highlighted in blue).

Q3: Are there potential adaptations of HADEMIF that could work with black-box models?

Thanks for your good question. We propose two feasible strategies for adapting our approach to black-box models:

• Knowledge Distillation: This strategy involves transferring knowledge from a black-box model to a student white-box model. The hidden states of the student model can then be passed into an MLP network for hallucination detection within the semantic space. While this approach may detect the hallucinations in the internal space of both student and teacher models, it offers a possible way to access the internal states of black-box models.
• Direct Logits Input: For a simpler adaptation, one can directly input the logits vector into the MLP network. Although the logits vector does not fully capture the LLMs’ internal states, it serves as an intermediary between the model’s internal and output spaces, reflecting, to some extent, the model’s internal states.

We will explore detailed approaches for applying our method to black-box models in future research.

[1] Out-of-distribution detection and selective generation for conditional language models.

[2] Uncertainty estimation in autoregressive structured prediction.

[3] Towards collaborative neural-symbolic graph semantic parsing via uncertainty.

Q4: What additional metrics could be integrated into the framework?

To strike a balance between efficiency and performance while enhancing the interpretability of D3T, our approach incorporates five types of representative metrics. Comparative and analytical experiments in our manuscript demonstrate their effectiveness. To further improve detection performance, additional metrics may be considered, such as the angles between hidden states and the discriminative weights in the final linear layer for each token, loss gradient, robustness, as well as other statistical properties (e.g., variance) of the logits and probability vectors. Our future research will investigate the incorporation of additional metrics and develop efficient methods for their computation, with the goal of further improving both detection performance and computational efficiency.

We have revised the sentences in the limitations section (Appendix A.10) of our manuscript, which now read as follows: "Additionally, to further enhance detection performance, future research could explore the integration of additional metrics, such as those related to learning dynamics (e.g., loss gradient) and prediction robustness, and develop efficient computational methods for their evaluation."

Q5: How does the framework impact user trust in LLMs?

Our approach enhances user trust in LLMs in two key aspects:

• Enhancing model calibration and decreasing hallucinations can enhance the users’ trust in the generations of LLMs. These two aspects can be quantified through a set of metrics that we have adopted. From Tables 1, 2, and 3 in our manuscript, we adopted four metrics, including ECE, Brier, acc@q, and cov@p, to assess the performance of model calibration and hallucination mitigation. The results demonstrate that the performance of our approach consistently surpasses that of the compared baselines. Moreover, we evaluated the hallucination detection performance of our approach using metrics AUROC and PCC. The results presented in Table 8 of our revised manuscript manifest the superior hallucination detection performance of our approach.
• The enhancement of interpretability can also improve user trust. Using D3T as the hallucination detection network helps to reveal the hallucination detection rules (Appendix A.7), showing how different combinations of characteristics determine the existence of hallucinations, and it can also uncover the importance of various prediction characteristics in the hallucination detection process (Section 4.6). This enhances decision transparency, providing users with a deeper understanding of the existence of hallucination in LLMs, thereby further increasing their trust in the model's decisions when applying it.

Q6: The integration of two separate networks may increase complexity.

We recognize that the computational complexity is increased, which is, in fact, an inherent challenge when incorporating additional modules, such as adapters in parameter-efficient fine-tuning of LLMs, as well as in the LITCAB method. Nevertheless, we have analyzed the complexity of the two hallucination detection networks in Appendix A.11, demonstrating that these networks are parameter efficient and exhibit strong scalability. Furthermore, we expanded the discussion on computational costs in Appendix A.12 of the revised manuscript. Our analysis reveals that using Llama2-7B as a case study, the time required to update the two hallucination detection networks per iteration is only 0.9% of the time needed to update the LLMs with LoRA (rank = 8).

Dear Reviewer ApSu:

We hope that the revisions and clarifications have effectively addressed your concerns. Your insightful feedback has greatly contributed to enhancing the quality of our paper. If any aspects remain unclear or if you have additional questions or suggestions, please feel free to let us know. We deeply value your comments and would be truly grateful if you could consider incorporating these improvements when reassessing our work.

Thank you once again for your time and effort in reviewing our work.

Sincerely,

Authors

Dear Reviewer ApSu:

We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript. As the author-reviewer discussion period comes to a close in the coming week, we kindly inquire if you are satisfied with the clarifications and revisions we have made. Your feedback on the current revision would be extremely valuable, and we would be truly grateful for any further suggestions you may have to provide us with an additional opportunity to refine our work.

Once again, we would like to express our heartfelt gratitude for your thoughtful and constructive review, which has been instrumental in strengthening our submission. Should any further concerns remain, we are fully committed to addressing them.

Best regards,

Authors

Dear Reviewer ApSu, we sincerely thank you for your thoughtful and thorough review of our manuscript, which has significantly strengthened the quality of our manuscript. As the discussion stage is ending soon, we would like to kindly inquire if you have any remaining questions or concerns. Any feedback is greatly appreciated and will undoubtedly help further enhance our work. Moreover, we would be truly grateful if you would consider increasing the score in light of the revisions and clarifications we have made, which were directly informed by your insightful feedback. Thanks again for your efforts in reviewing our paper!