TruthFlow: Truthful LLM Generation via Representation Flow Correction

We thank the reviewer for their valuable feedback and constructive suggestions.

---

Q1: Statistical evidence A1: Thank you for your suggestion. We further conduct the following statistical analysis to demonstrate this limitation. Specifically, we calculate the cosine similarity between the universal vector $y$ and each specific truthful correction vector $x_i$ obtained from LLM internal states. We first calculate the variance of the cosine similarities $\frac{\langle x_i, y\rangle}{\left\|x_i\right\|_2\left\|y\right\|_2}$ and find it to be 0.536. Considering the range of cosine similarity, this suggests a quite high statistical variance. Furthermore, we also visualize the distribution of the cosine similarities in the following URL to show the truthful correction vector diversity. https://anonymous.4open.science/r/13040_Rebuttal-432A/

Q2: Transferability A2: Thanks for your suggestion.

• First, we emphasize that there's no quality drop in our transfer experiment (only the improvement is less significant). This is perfectly normal as transferring across different datasets is indeed difficult and our method has already achieved better performance compared with baselines.
• We understand that the reviewer asks for a clearer picture of the transfer performance. However, the full transfer performance matrix may not be a good idea and is not adopted in prior related works. The reason is that for data-driven methods such as TruthFlow, the transfer performance not only depends on the method itself but also on the quality of training data. Specifically, we find TruthfulQA more suitable for eliciting truthful answers because of the elaborate "traps" in the questions. The truthful correction vectors obtained under such a scenario may capture more truthful-related information. In contrast, questions in HaluEval, NQ, and TriviaQA are more related to specific knowledge. If the LLM lacks sufficient knowledge, the representations of correct and incorrect answers won't constitute a strong contrast in truthfulness. As pointed out by Reviewer P7qz, existing work [1] has shown that even if directly training intervention methods on datasets such as NQ, it may lead to incremental performance gain, or even drop compared to the base model, not to mention the transfer case.
• To provide more evidence for transferability, we transfer TruthFlow to MedHallu [2]. The results show that TruthFlow can also be generalized to mitigate medical hallucinations by achieving improved performance compared to the base LLM.

[1] Liu et al., Enhancing Language Model Factuality via Activation-Based Confidence Calibration and Guided Decoding. EMNLP 2024

[2] Pandit et al., MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models. arXiv preprint

Q3: Evaluation on other datasets for real-world generalization A3: To test real-world generalization, we conduct experiments on a text summarization and a medical hallucination QA task. Please see our reply to Reviewer P7qz's Q3. The performance gains indicate that TruthFlow can generalize to more practical tasks. Besides, to check the general utility, we evaluate TruthFlow on MMLU. Please see our reply to Reviewer 1fpP's Q1. We observe a minor decrease which shows that TruthFlow doesn't hurt general utility while improving factuality.

Q4: Selection of $\alpha$. A4: We apologize for the confusion.

• In practice, we reserve a small part of the training set for hyperparameter tuning. Specifically, we conduct a grid search on $\alpha$ from $[0.5, 4.5]$ and pick the optimal one on the validation set.
• To ensure a fair comparison, we conduct similar hyperparameter tuning on other baselines. For example, on ITI and NL-ITI, we conduct a grid search on the intervention intensity from $[1, 25]$. On AD, we tune the info layer ranging from 22 to 30 for 32-layer LLM and 28 to 38 for 40-layer LLM and tune the entropy control penalty coefficient $\lambda$ from $[0.4, 1]$.
• Note that for transferability, since the flow model is trained on the original dataset, using the same $\alpha$ for other datasets is usually too large and can easily get overfitted. Thus our hope is to keep the ratio of the query representation norm and correction vector norm in a stable position. Specifically, we calculate the average query representation norm as $n_q$ and the average correction vector norm as $n_c$ and set $\alpha = \beta* n_q/ n_c$ where $\beta$ is a constant parameter (e.g., 0.1) and rounded to the nearest half-integer values.

Q5: Typos and failure cases A5: Thanks for pointing out. We will fix the typos in the revision. Due to space constraints, we show some failure cases in https://anonymous.4open.science/r/13040_Rebuttal-432A/. We observe that some failures arise from the lack of necessary knowledge, which is consistent with the analysis in [1].

We thank the reviewer for their valuable feedback and constructive suggestions. We address the questions as follows:

---

Q1:Slightly lower Info score.

A1: We actually observed and analyzed this phenomenon in the paper's “Qualitative Study” paragraph before section 5. Specifically, we find that some of the best answers in TruthfulQA are labeled as less informative, (e.g. the best answer is “I have no comment”). And since TruthFlow makes the model output more truthful, it inevitably causes the Info score to be lower on those questions (see an example in Table 2). This leads to the overall Info score being lower. Please note that TruthFlow can still provide informative answers to other questions whose correct answer is indeed informative.

Q2: Additional domains, such as legal or medical fields.

A2: Thanks for your suggestion. We conduct additional experiments to apply TruthFlow on the medical QA task. Specifically, We test on MedHallu [1] dataset which contains 1000 human labeled medical questions, along with knowledge, ground truth, and hallucinated answer. Due to time constraints, we only compare TruthFlow with the base LLM and ITI. We follow the evaluation metric in TruthfulQA to calculate the True score, Info score, and True*Info score using GPT-4o. The results are listed below. We can observe that TruthFlow still achieves significant improvement over the base model and ITI despite a slight decrease in the Info score.

[1] Pandit et al., MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models. arXiv preprint

We thank the reviewer for their valuable feedback and constructive suggestions. We address the questions as follows:

---

Q1: Lack formal theoretical proofs for some claims, such as the projection onto a "truthfulness subspace"

A1: First, we would like to emphasize that we do not claim theoretical contributions in this work. This is an empirical work to provide strategies for mitigating hallucinations in LLMs. But since you asked, we would try our best to show that the idea of the truthful subspace is more than simply our assumption despite that it is hard to formally prove its existence.

Intuitively, the top singular vectors of the matrix correspond to the main basis directions that can point from hallucinated states to truthful states. From this perspective, calling the subspace of these top singular vectors a "truthfulness subspace" is reasonable. Existing works such as [1] also apply similar approaches like SVD to identify "toxic subspace" and justify the identified subspace with theoretical insights. Of course, what is the accurate number of top singular vectors to make this subspace still remains a mystery. We set it as a hyperparameter now but we would like to explore more on this in our future work.

[1] Uppaal et al., Model editing as a robust and denoised variant of DPO: A case study on toxicity. ICLR 2025

Q2: Conclusion for Table 4? Which layer should be chosen?

A2: The direct conclusion from Table 4 is that for our method, intervention at the 12th layer is the best compared to the other layers. Note that prior works (such as TruthX [2] and BiPO [3]) all suggest that intervention at the intermediate layers of the model typically leads to the best results. This aligns with our findings in Table 4.

In practice, the choice of intervention layer is more of an empirical hyperparameter and may change due to the LLM architecture, the data, the specific intervention method, etc. However, we notice that the best layer is relatively consistent in models of the same series with similar parameter amounts.

[2] Zhang et al., TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Space. ACL 2024

[3] Cao et al., Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization. NeurIPS 2024

Q3: What about the effect of applying the transition across multiple layers simultaneously?

A3: Thank you for your suggestion. We conduct experiments to apply the transition across two layers simultaneously on Llama-3-8B-Instruct model. Specifically, we extract hidden states from the two selected layers and concat them to form a larger vector to train the flow model. We test some layer combinations on the TruthfulQA open-generation task and report the results in the table below. Applying intervention across two layers simultaneously may slightly (not always) improve the single-layer intervention, but the performance gain is not very significant. We will leave this multi-layer TruthFlow as our future work.

Q4: The lack of human evaluation.

A4: Thank you for your suggestion. Certainly, human evaluations would make the results more convincing. Yet our experimental design follows most existing works in this direction and we believe it is enough to justify its better performance. We (the research team) also manually check the generated results and feel consistent with our numerical results. Due to time limitations, we would leave human evaluation as our future work.

Q5: Reason for slightly lower Info metric.

A5: We actually observed and analyzed this phenomenon in the paper's “Qualitative Study” paragraph before section 5. Specifically, we find that some of the best answers in TruthfulQA are labeled as less informative, (e.g. the best answer is “I have no comment”). And since TruthFlow makes the model output more truthful, it inevitably causes the info score to be lower on those questions (see an example in Table 2). This leads to the Info score being lower. Please note that TruthFlow can still provide informative answers to other questions whose correct answer is indeed informative.

We thank the reviewer for their valuable feedback and constructive suggestions. We address the questions as follows:

---

Q1: General utility. A1: Thank you for your suggestion. We conduct experiments to test the general utility on MMLU. We evaluate Llama-3-8B-Instruct and TruthFlow on the whole 57 subjects of MMLU in a 5-shot prompt setting. The results presented in the table below indicate that the MMLU accuracy with TruthFlow shows only a minor decrease of 0.2%, which suggests that TruthFlow does not hurt the LLM's general utility while improving factuality.

Q2: How to get the blue arrow in Figure 2? A2: We apologize for the confusion. In short, the blue arrow is the average over all light blue arrows. Specifically, we first extract truthful and hallucinated states and reduce the dimensionality. We denote truthful data points as $\{ (x_i^t, y_i^t ) \}_{i=1}^N$

and hallucinated data points as $\{ (x_i^h, y_i^h ) \}_{i=1}^N$

where $N$ refers to the number of data points and $x\in\mathbb{R}, y\in\mathbb{R}$ as the coordinate values of x-axis and y-axis. Then we calculate the mean point $\left(\bar{x}^t, \bar{y}^t\right) = \left(\frac{1}{N}\sum_{i=1}^N x_i^t, \frac{1}{N}\sum_{i=1}^N y_i^t\right)$ and $\left(\bar{x}^h, \bar{y}^h\right) = \left(\frac{1}{N}\sum_{i=1}^N x_i^h, \frac{1}{N}\sum_{i=1}^N y_i^h\right)$. The blue arrow is from the mean hallucinated data point to the mean truthful data point, $\left(\bar{x}^t - \bar{x}^h, \bar{y}^t - \bar{y}^h\right).$

Q3: Compare with post-training baselines such as GRATH A3: Thank you for your question. Please note that TruthFlow is a representation intervention method, which only involves the inference stage of LLMs, with no additional “post-training” on LLMs. While methods such as GRATH require additional training/finetuning of LLMs which are much more costly compared with inference-based solutions. Therefore, it is unfair to compare with post-training methods if not counting the cost. Furthermore, GRATH actually requires an iterative training schedule with updated training data, which makes it even harder to compare. Nevertheless, we still conduct experiments to compare our TruthFlow with GRATH using their publicly released model trained for ten iterative DPO rounds. We report the results in the table below and it achieves very similar performance as TruthFlow. Please note that this particular GRATH model is trained upon ARC-challenge data (much larger than TruthfulQA) and part of TruthfulQA data for over 10 hours, which is much more costly.

Q4: Importance of truthful projection. Effect of increasing the number of top singular vectors. A4: Thanks for your questions.

• We have actually conducted ablation studies on truthful projection in Table 5 of Section 5.4. We found that without projection, TruthFlow still has an overall truthful performance gain over base LLMs. After applying projection, the truthfulness and informativeness are further improved.
• Figure 3 shows that when the number of top singular vectors is small (e.g., 5), the performance is not good enough due to the severe loss of information. On the other hand, when we choose a larger number of top singular vectors, it gradually becomes sufficient to capture the truthful subspace and thus leads to much more stable performances. At this point, further increasing the number of singular vectors would not help much with the overall performance.

Q5: Replacing flow matching model with any generative models A5: Thanks for the interesting question. We believe that general diffusion models are not a good choice since they typically start generating from random Gaussian noise (maps from Gaussian distribution to the target distribution). Our TruthFlow leverages the flow matching model to capture the distribution trajectory from the query representation distribution (which is not Gaussian) to the correction vector representation distribution. In short, if the generative model can build the trajectory between any two distributions, it may work here. If it maps from Gaussian to the target distribution, it is not suited here.

We thank the reviewer for the valuable feedback and constructive suggestions. We address the questions as follows:

---

Q1: New reference.

A1: We appreciate the reviewer for pointing out the reference [1]. We will discuss and cite this work in our revision. One thing to clarify is that we test other datasets for transferability while [1] trains their method on various datasets. Therefore, our case is actually more challenging and smaller performance gain under transferability experiment is reasonable. [1] Liu et al., Enhancing Language Model Factuality via Activation-Based Confidence Calibration and Guided Decoding. EMNLP 2024

Q2: Theoretical justifications for why flow matching is suited.

A2: From our analysis in Sec 3.2, we hope to obtain a query-specific solution that requires us to build a mapping from the query representation distribution to the corresponding correction vector distribution. Furthermore, this solution should also be efficient and well-generalizable, to make it practical to use in LLM inference situations. Flow matching is well-known for building a linear trajectory from source to target distributions, which satisfies our need to capture the distribution mapping. Since the trajectory is linear, it's easier/faster to sample and generalize well. Thus we believe flow matching is well-suited for our goals here. We hope this addresses your concern.

Q3: Tasks beyond factual QA.

A3: Note that it is non-typical for factuality-related works to test dialogue and summarization tasks (since these tasks focus more on how LLMs handle and understand context rather than test truthfulness). However, we have tried to train TruthFlow on XSum [2]. Due to time constraints, we randomly sample a subset of data (same size as TruthfulQA) to test its performance. Considering our training process requires pairwise data but there's no "incorrect summary" in XSum, we use GPT-4o to generate seemingly plausible but incorrect summaries for training data. We use the commonly used ROUGE metric for XSum to evaluate. The results suggest that TruthFlow achieves significant improvement upon the base LLM for summarization.

In addition, we train TruthFlow on medical hallucination benchmark [3] and evaluate following the same set of evaluation metrics (True, Info, and True*Info scores). The results are presented in the table below. Although slight decrease in Info, TruthFlow outperforms base LLM and ITI in True and True*Info, demonstrating the effectiveness of TruthFlow in medical domain hallucinations.

[2] Narayan et al., "Don’t give me the details, just the summary." Topic-Aware Convolutional Neural Networks for Extreme Summarization. EMNLP 2018

[3] Pandit et al., MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models. arXiv preprint

Q4: Hyperparameter sensitivity.

A4: Model Capacity. We conduct an ablation study using three sizes of the model capacity (by adjusting "depth" and "feature_scale"). The size of the small, middle, and large network is 0.05B, 0.11B, and 0.2B in bytes. We test them on TruthfulQA. When the neural network is small, it has difficulty fitting the training data, leading to generating query-specific truthful vectors of lower quality. In comparison, when the model capacity is large enough to fit the training data, the performance becomes stable. The additional parameters do not largely improve TruthFlow-L over TruthFlow-M.

Training data size. We conduct an ablation study using a random subset of TruthfulQA training data. We find that using 1/2 of the original training data leads to worse performance on open-generation task and using 1/4 of training data leads to a more significant performance drop.

Choice of subspace. We have already conducted ablation studies on the number of top singular vectors in Figure 3, which determines the subspace. In particular, selecting too many singular vectors generally means retaining most information but could also keep the noisy or hallucinated information while selecting too few singular vectors could lead to severe loss of information.