On-Policy Fine-grained Knowledge Feedback for Hallucination Mitigation

Thank you for your constructive feedback. For the mentioned weakness, we want to provide further clarification:

Weakness 1: Although our proposed method cannot completely resolve the hallucination problem—since theoretically, as noted in [1], it cannot be entirely eliminated—its primary aim, much like most current work [2, 3], is to mitigate hallucinations. Compared to other works, the RLFH algorithm leverages the fine-grained reward information provided by the LLM itself to achieve on-policy hallucination mitigation. Unlike some other methods that utilize off-policy approaches, the on-policy characteristic of RLFH better aligns the model's factual performance with its internal knowledge boundaries. For instance, regarding the evasive ignorance you mentioned, ideally, an on-policy algorithm would encourage the model to provide content it is most confident about, as this increases the expected overall reward.

• [1] Hallucination is Inevitable: An Innate Limitation of Large Language Models
• [2] Fine-tuning Language Models for Factuality
• [3] FLAME: Factuality-Aware Alignment for Large Language Models

Weakness 2: When verifying the factuality of responses, RLFH does not rely solely on the original answer to the question. Instead, it can also utilize external knowledge sources for further validation (in our case, we use the Wiki knowledge base). When the external knowledge base does not cover the information contained in the response, we require the model to label the information as "Vague," indicating that its accuracy cannot be determined. Such information receives a lower reward than "Correct" information but a higher reward than clearly "Wrong" information. This approach encourages the model to avoid generating ambiguous information that is difficult to assess for accuracy. We believe that such content is also a form of hallucination.

Weakness 3: Our algorithm does not impose any competency requirements on the policy itself. However, for the reward model, we need it to possess some level of instruction-following capability to conduct factual verification according to our predefined prompts. As discussed in Section 4.5, we found that more capable models perform this task better. Furthermore, when considering the relationship between the capabilities of the policy and the reward model, we believe that our approach does not require the reward model to be more capable than the policy. This is because the reward model can rely on external knowledge for factual verification, which is a relatively straightforward task.

Question 1: Apologies for the confusion. We have rectified this issue in the new version of the paper.

Dear Reviewer,

Considering the rebuttal deadline is approaching, we sincerely hope to receive your response. If you have any further questions or concerns regarding our explanation, please do not hesitate to contact us. Your response is of great significance to us for improving this work, and we look forward to hearing from you.

Thank you for your constructive feedback. For the mentioned weakness, we want to provide further clarification:

Weakness 1: Although RLFH relies on LLM to provide reward signals, it can retrieve external knowledge for factual verification. This task is relatively straightforward for LLM. Additionally, even if a model's response cannot be verified by external knowledge, the LLM can still indicate this by labeling it as "Vague." This ensures that the reward model's signal remains reliable. On the other hand, as an on-policy algorithm, RLFH does not directly inject knowledge into the model but rather corrects the model's use of existing knowledge, reducing the risk of error accumulation. This is one of the advantages of RLFH compared to some other works [1,2].

• [1] Fine-tuning Language Models for Factuality
• [2] FLAME: Factuality-Aware Alignment for Large Language Models

Weakness 2: As shown in Table 1, our method shows improvement over FACT [1], a DPO-based RLHF algorithm which is specifically designed for hallucination, on both ID and OOD evaluation.

• [1] Fine-tuning Language Models for Factuality

Weakness 3: Thank you for your reminder. We have made the revisions in the latest version of the paper.

Dear Reviewer,

We sincerely appreciate your comments, as well as the time you've dedicated to reviewing our work. With the rebuttal deadline approaching, we look forward to any further feedback you might have. Please feel free to reach out with any additional questions or concerns.

Thank you for your helpful feedback. For the mentioned weakness, we would like to provide further clarification:

Weakness 1: In selecting baselines, we primarily chose two categories: 1) several advanced aligned models that typically have not undergone specific hallucination mitigation training, making them suitable for demonstrating the effectiveness of hallucination mitigation algorithms; and 2) other hallucination mitigation methods, such as ITI, DOLA, and FACT. We implemented these algorithms using Vicuna-7B and compared them with RLFH, which is also based on Vicuna-7B. Our results demonstrate that, under the premise of a consistent model architecture, our algorithm outperforms other hallucination mitigation algorithms.

Weakness 2: To further demonstrate the improvements of our algorithm, we incorporated FAVA [1] as an additional hallucination metric. Unlike FactScore, FAVA detects various types of hallucination errors based on reference documents. We report the average number of errors per sentence as a measure of hallucination:

Compared to FACT and other baselines, RLFH significantly reduces the average error rate, as indicated by errors per sentence. This further validates the effectiveness of our algorithm.

While metrics like FactScore and FAVA focus on hallucination, some hallucination mitigation strategies, such as knowledge editing [2] or training-based methods [3], can inevitably compromise the model's core capabilities. In contrast, RLFH employs an on-policy framework with fine-grained rewards, allowing it to tailor model behavior precisely while minimizing the impact on other capabilities. Below is the evaluation on the Math and Code datasets:

As shown in the table, the RLFH algorithm reduces hallucinations without compromising the performance of the original model. In contrast, FACT reduces performance in areas beyond hallucination, such as Math (evaluated with GSM8K) and Code (evaluated with HumanEval), highlighting an additional advantage of our algorithm.

• [1] Fine-grained Hallucination Detection and Editing for Language Models
• [2] SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
• [3] Fine-tuning Language Models for Factuality

Weakness 3: Although RLFH relies on external knowledge sources for factual verification, we believe this is an important feature to ensure reliable hallucination mitigation. Moreover, as an alignment algorithm, RLFH does not inject new knowledge; instead, it adjusts how the model utilizes its existing internal knowledge. Therefore, in scenarios where external knowledge validation is not possible, we can train the alignment on datasets that allow for such validation and generalize to scenarios without external knowledge bases. The feasibility of this approach is demonstrated in Table 1, where we trained our model on HotpotQA, and its factual accuracy improvements generalized to the out-of-domain test sets SQuADv2 and Biography.

Question 1: Thank you for your question. We address this in our response to Weakness 1.

Dear Reviewer,

We are truly grateful for the feedback you have provided on our work. We have tried our best to address your concerns. With the rebuttal deadline approaching, we are eager to receive any further comments you may have. Please do not hesitate to contact us if you have any additional questions regarding our responses.