FLAME : Factuality-Aware Alignment for Large Language Models

1. Re: No external baselines are used in the comparison.

We thank you for the suggestion. As far as we know, the existing best approach to factual alignment is the method introduced by Tian et.al. [1]. Note that, this approach conducts factual alignment directly on the target task (e.g., biography generation) while we conduct factual alignment on more diverse and general instruction following tasks. Although it is hard to conduct a direct comparison with the baseline, as mentioned in line 277, we do provide a baseline which mirrors the approach by Tian et.al. [1], $\mathrm{SFT }+ \mathrm{DPO}^{\textrm{fact}}$ as shown in row 3 of Table 3, where factuality is the only objective of DPO. The main purpose of our paper is not to propose a state-of-the-art approach for factual alignment. Instead, we find the side effect of applying factual alignment to instruction tuning and manage to mitigate the side effect.

2. Re: It will be great to conduct human evaluations even just on a few examples.

Thanks for the suggestions. We agree that adding human evaluations would make the result more convincing. We have conducted manual evaluations for two baselines and FLAME on a few representative cases from Alpaca Eval and Biography in case study, as shown in Appendix Figure 10. We will make it clear in the section of case study.

3. Re: Low-level details like few-shot prompts, and implementation of fact decomposition are omitted.

Thanks for the suggestions. Below are the low-level details about our few-shot prompts and fact decomposition, which we will add into our revised manuscript.

For the detail of few-shot prompts, as mentioned in lines 195--198, for each instruction $x$, we retrieve the top-5 similar instructions ($x_0 \cdots x_4$) along with the corresponding human responses ($\mathrm{Human}(x_0) \cdots \mathrm{Human}(x_4)$) from OASST (our seed data with 3.2K instruction--human response pairs). Then, concatenating them as the few-shot prompt $x_4, \mathrm{Human}(x_4), x_3, \mathrm{Human}(x_3), \cdots, x_0, \mathrm{Human}(x_0), x$ to elicit the response from the pre-trained LLM.

For the implementation of fact decomposition, we reuse the code from FActScore to conduct fact decomposition but replace the GPT3.5 with our fine-tuned Llama2 7B model on the public datasets (i.e., RoSE benchmark [2], CLAIMDECOMP [3] and EXPERT QA dataset [4]). Note that the purpose of fine-tuning our Llama2 7B for atomic fact decomposition is to save the cost of creating factuality preference training data.

4. Re: In the pilot experiment, doing DPO with FS seems to work reasonably well. Have you tried similar approaches in the real experiments?

DPO with FS can be considered the approach by Tian et.al. [1] and as mentioned in point 1, in our main experiment for instruction tuning, we do provide a baseline as shown in row 3 of Table 3, which is mentioned in line 277. We will make the statement more clear in the revised version.

[1] Fine-tuning Language Models for Factuality. https://arxiv.org/abs/2311.08401

[2] https://github.com/Yale-LILY/ROSE

[3] https://github.com/jifan-chen/subquestions-for-fact-checking

[4] https://github.com/chaitanyamalaviya/ExpertQA

1. Re: This approach may struggle with instructions that the original model cannot generate factual answers for.

We thank you for the insight. This is exactly what we found in the paper; that is, it is challenging to teach LLMs to learn new knowledge in the finetuning stage. Inevitably, forcing LLMs to learn new knowledge (e.g., learning from RAG or human written response) may cause more hallucination, which is also found in the concurrent paper [1]; thus, we propose to fine-tune LLMs only on its own generated responses. We agree that it is worth conducting exploring how to inject new knowledge to LLMs during fine-tuning stage without any side effects but this is orthogonal to our work. We will make our claim more clear in our revised manuscript.

2. Re: The proposed strategy relies on accurately identifying the instruction type initially, which is limited by the model's ability to correctly classify the instruction type.

Yes, we admit that FLAME does require a module to distinguish the types of instructions. However, instead of treating it as a weakness of our approach, we believe that it is the important finding of our paper. That is, not all instructions require factual responses, and overly optimizing for factuality could lead to degradation in instruction following capability (condition 3 vs 4 in Table 3). Furthermore, the previous study [2] also find that different instructions require different evaluation strategies. Finally, we also demonstrate that the task of classifying the instruction type can be easily done by an instruction following fine-tuned model rather than a specially fine-tuned model.

3. Re: In the pilot study, it is unclear whether the $\mathrm{PT}$ and $\mathrm{PT}^{\text{\tiny RAG}}$ are evaluated using the same protocol as other methods.

Yes, in our pilot study $\mathrm{PT}$ and $\mathrm{PT}^{\text{\tiny RAG}}$ are evaluated using the same protocol as other methods. However, this does not mean the results contradict our claim: fine-tuning LLMs on their own generations appears to be crucial for factual alignment. First of all, the responses from $\mathrm{PT}$ and $\mathrm{PT}^{\text{\tiny RAG}}$ are generated using 5-shot demonstrations while the responses from SFT and DPO models (only fine-tuned on 495 instructions for biography generation) are generated with 0 shot. Especially, $\mathrm{PT}^{\text{\tiny RAG}}$ also concatenate retrieved evidence; thus, it is reasonable that $\mathrm{PT}^{\text{\tiny RAG}}$ shows far better factual accuracy. Although SFT (fine-tuned with a few examples) zero-shot generation shows a slight factuality degradation compared to $\mathrm{PT}$ 5-shot prompts, we do see that the DPO model improves over the $\mathrm{PT}$ model in Table 1 (condition 5 vs 1). To be clear, our claim comes from the observation that SFT and DPO fine-tuned with LLM’s own generation versus RAG generation (3 vs 4, 5 vs 6) even though RAG generation is shown to be far more factual.

4. Re: SFT and DPO do not improve models’ factuality over 5-shot prompts.

We want to clarify that in our paper, improving LLMs’ factuality does not mean that we can make the instruction following fine-tuned LLMs generate more factual responses than the pre-trained LLMs with few-shot demonstrations. In fact, there are previous papers which find that instruction tuning or alignment may degrade LLMs’ performance on the standard knowledge benchmarks, called alignment tax [3]. The alignment tax can explain why our DPO fine-tuned model (only on biography generation) can improve over $\mathrm{PT}$ (Table1 condition 5 vs 1) but the DPO fine-tuned model (on diverse instruction following tasks) cannot improve over $\mathrm{PT}$ (condition 7 in Table 3 vs condition 0 in Table 2). Again, we want to highlight that the main purpose of the paper is to study why instruction following fine-tuning tends to cause hallucination and how to mitigate the issue rather than addressing the alignment tax. Thanks for the insightful question, we will add further explanation in our revised paper.

5. Re: As discussed in Sec 5.5, conducting fact checks and computing factuality rewards solely for fact-based sentences can lead to more factuality errors. Clarification is needed on how FS is calculated for the experiments in Sec 5.2 and 5.3.

All our FActScore evaluation is based on the assumption that all the sentences in the responses are fact-based ones for two reasons. First, as pointed out in our ablation study, excluding the non-fact-based sentences may cause noise. Second, from our eyebrow check, LLMs’ generation for Biography, Alpaca Fact and FAVA datasets are mostly fact-based sentences. Figure 10 in Appendix showcases a few LLMs’ responses for Biography and Alpaca Fact.

[1] Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? https://arxiv.org/abs/2405.05904

[2] Branch-Solve-Merge Improves Large Language Model Evaluation and Generation. https://arxiv.org/abs/2310.15123

[3] Training language models to follow instructions with human feedback https://arxiv.org/pdf/2203.02155

1. Re: The baselines compared in this work are limited to different settings of SFT and DPO only.

We thank your suggestion to compare with the baseline from Tian et.al. [1]. First of all, we want to clarify that Tian et.al. [1] mainly focus on fine-tuning LLMs on a specific task (e.g. biography generation) while FLAME focuses on more general instruction following tasks, where biography generation is a subtask for us to evaluate factuality. Thus, it is hard to make a fair comparison between the two approaches. Nevertheless, as we mention in Section 5.3 (line 277), we do apply the approach from Tian et.al. [1] to our alignment training as $\mathrm{SFT}^{\textrm{fact}}$ and $\mathrm{DPO}^{\textrm{fact}}$, where factuality is the only optimization objective. This result (row 4 vs 3 in Table 3) supports our claim from related work (lines 84--85) that solely focusing on factual alignment may impact LLMs’ instruction following capability. Furthermore, we also point out the importance of finding those fact-based instructions among the instructions, which is ignored by Tian et.al. [1] if we only conduct factual alignment on biography generation tasks. In other words, FLAME is the extension of Tian et.al. [1] to more general instruction tuning tasks. We will revise the manuscript to clarify that we have done our best to compare with the existing factuality fine-tuning method [1] in our main experiment.

2. Re: The paper may need to add explanations of the differences between these methods to clarify its own novelty.

We thank you for the good references and do agree that we need to compare the more recent work in our related work [2][3]. The main contribution of the related work[2][3] focuses on using the LLM itself as the factuality judge to enhance its factuality. Similar to Tian et.al. [1], they mainly focus on fact-based instructions and can be considered as an improved version of FactTuneMC from Tian et.al. [1]; thus, ignoring the impact of factuality alignment on LLMs’ instruction following capability. Furthermore, the proposed factuality alignment approaches from [2][3] can be integrated into FLAME to create more accurate factuality pairs. We will add the comparison and cite the references in our revised version.

3. Re: In comparison to training, have the authors considered comparing representation editing baseline methods?

We thank you for the suggestions. Although representation editing baseline approaches (e.g., DoLa [4] and ITI [5]) also show promising results, these approaches rely on a development set of the target task to tune the hyperparameters and focus on a single evaluation metric. As we have shown in our main experiment, focusing only on improving factuality may sacrifice models’ instruction following capability. Furthermore, as shown in Tian et.al. [1], the representation editing approaches are not more effective than fine-tuning for factuality, which is one of our baseline approaches (row 3 in Table 3).

4. Re: Could the authors supplement experiments on the TruthfulQA-MC in Table 4 to provide a measure of multi-choice performance?

We thank you for the suggestion. Below is the comparison of TruthfulQA-MC (zero-shot). Note that, we do not observe significant differences between models on TruthfulQA-MC task. This is possibly because we mainly focus on the tasks of long-form response generation while TruthfulQA-MC task is formed by short-form answers. The discrepancy between improving LLMs’ factuality on long-form and short-form generation is also found by the previous work [4]. We will add the experiment and explanation to our revised manuscript.

Reference:

[1] Fine-tuning Language Models for Factuality. https://arxiv.org/abs/2311.08401

[2] Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation. https://arxiv.org/abs/2402.09267

[3] GRATH: Gradual Self-Truthifying for Large Language Models. https://arxiv.org/pdf/2401.12292

[4] DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models. https://arxiv.org/abs/2309.03883

1. Re: Model Size and Generalizability. It would be beneficial to investigate whether FLAME's effectiveness extends to smaller models, such as 7B or even smaller, given that the factuality-aware SFT relies on self-supervision through few-shot prompting.

We thank you for the helpful suggestion. We believe our finding is also valid for smaller models. In our pilot study, the condition of row 3 in Table 1 is equal to factuality-aware SFT with self-supervision through few-shot prompting. The pilot study on smaller models motivates us to extend our study to more general instruction tuning rather than focusing on biography generation. However, we have to admit that it is more challenging to fine-tune a smaller model (e.g., Llama2 7B) to follow instructions, especially that we only use 3200 SFT samples from OASST dataset. Not to mention applying the self rewarding [1] methodology to improve the models’ instruction following capability. Note that we choose not to rely on the large size and high quality SFT data (e.g., Alpaca cleaned dataset or UltraFeedback) generated from GPT4.

2. Re: Factuality Evaluation

Although FActScore is the best automatic factuality evaluation tool we can get access to, we agree and discuss the limitation of FActScore in the Section of Limitation. We thank you for referring to the more recent factuality evaluation tool (e.g., Veriscore).

3. Re: Multi-faceted Evaluation

We thank you for the helpful suggestion. Below is the comparison between the baseline and factuality-aware fine-tuned models on MMLU and GSM8K. Their results are very close. We will add the evaluation on standard benchmark in the appendix.

4. Re: While FLAME primarily focuses on DPO, can it also be applied to conventional reinforcement learning from human feedback (RLHF) methods like PPO?

We thank you for the question. Yes, we believe FLAME can be also applied to PPO. As shown in our ablation study (the third row in Table 6), we can combine the rewards of instruction following capability and factuality into a single scalar reward. With the single scalar reward, we can conduct PPO for preference fine-tuning. However, due to the complexity and instability of fine-tuning with PPO, we instead use DPO as a proof-of-concept in our experiment. We will add this discussion in Section 5.5 in the revised manuscript.

5. Re: Are there plans to release the code and models trained using FLAME for the research community to replicate your methods?

Thanks for the suggestions. We will consider releasing the code or the scripts for creating the preference training data.