Is Factuality Enhancement a Free Lunch For LLMs? Better Factuality Can Lead to Worse Context-Faithfulness

Weakness 8: The model internal analysis part is great and potentially useful. I wonder if the authors could offer some crisp conclusions/takeaways in addition to presenting some numbers and figures...

We appreciate the reviewer's suggestion to provide clearer conclusions from the interpretative analysis section. Below is a concise summary of the key findings:

(1) Factuality enhancement methods reduce the probability of new knowledge tokens while increasing the probability of parametric knowledge tokens.

(2) These methods shift the distribution of new knowledge tokens toward lower probability ranges, indicating diminished confidence in in-context knowledge compared to the baselines.

(3) Context injection alters the distribution of knowledge representations, causing them to deviate from their original factuality.

(4) Factuality enhancement methods limit the natural dispersion of hidden states after context injection. Consequently, the output decoded from these constrained hidden states remains more aligned with the original parametric knowledge, rather than effectively incorporating new contextual knowledge as seen in the baseline.

Insights for Future Work

Based on the findings above, we propose the following directions:

<1> Logits Distribution Adjustments (Findings 1 & 2): Since factuality enhancement methods amplify the probabilities of parametric knowledge tokens, making it harder for in-context knowledge tokens to surpass them, a "patching" strategy (as suggested in Weakness 7) could be employed to boost the probabilities of context tokens, thereby improving context faithfulness.

<2> Hidden State Distribution Control (Findings 3 & 4): Factuality enhancement methods restrict the dispersion of hidden state distributions, which limits the model's ability to integrate new contextual knowledge. Future work could explore controllable model editing approaches [7] to regulate this restriction, enabling broader and more flexible hidden state distributions even after factuality enhancements.

We believe these insights provide valuable guidance for addressing the trade-offs between factuality enhancement and context faithfulness in future research. Thank you for raising this important point!

Your suggestions for our work are invaluable, and we have greatly benefited from them—not only for our current research but also for guiding our future directions. Your feedback reflects your deep knowledge and professionalism in this field, and we would be delighted to continue engaging in discussions with you.

We have incorporated your suggestions to improve the quality of our manuscript and sincerely hope our responses enhance your impression of our work. We look forward to hearing your thoughts.

Once again, thank you very much for your time and thoughtful feedback.

[5] Shi, Weijia, et al. "Trusting Your Evidence: Hallucinate Less with Context-aware Decoding." NAACL 2024.

[6] Bi B, Liu S, Mei L, et al. Decoding by Contrasting Knowledge: Enhancing LLMs' Confidence on Edited Facts[J]. arXiv preprint arXiv:2405.11613, 2024.

[7] Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt. Nips 2022

[8] Meng, K., Sharma, A. S., Andonian, A., Belinkov, Y., and Bau, D. (2022b). Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229.

We sincerely thank Reviewer oQUh for raising the question about the scope of our definition of "Factuality Enhancement." Your insightful comments have prompted us to further reflect on and refine our approach.

Our response is organized into three parts to address both your primary concern and the related weaknesses you highlighted. Specifically, we:

(1) Clarify the original definition of "Factuality Enhancement" as presented in the manuscript and provide an explanation for the associated weaknesses you identified.

(2) Present supplementary experiments and analyses inspired by your suggestions, including evaluations using prompt-based, DPO-based, and SFT-based methods, which expand the scope of our study.

The Scope of "Factuality Enhancement"

This work highlights the significant negative impact that existing Factuality Enhancement methods may have on the context faithfulness of LLMs, supported by extensive experiments and analyses. In our submitted manuscript, "Factuality Enhancement" specifically refers to enhancing the factuality of LLMs by improving their ability to utilize their parameterized knowledge during the inference stage. As stated in our paper (pages 49-52): “... focuses on narrowing the gap between 'knowing' and 'telling' in LLMs to improve LLMs’ factuality.” This definition deliberately focuses on inference-stage enhancements, specifically addressing methods such as editing internal representations and adjusting the logits of tokens during decoding, without involving training or prompt engineering.

From a new perspective, knowledge conflicts can generally be categorized into inter-memory conflicts and context-memory conflicts [1]. Factuality Enhancement methods aim to improve the factuality of LLMs without introducing new knowledge (i.e., no additional training), addressing inter-memory conflicts. However, our analysis reveals that these methods can exacerbate context-memory conflicts, significantly diminishing the context faithfulness of LLMs. This phenomenon severely impacts the effectiveness of critical tools such as retrieval-augmented generation (RAG). We believe this finding contributes to the community’s understanding of the trade-offs inherent in LLM capabilities.

All authors agree that the reviewer's suggestion to expand the scope of Factuality Enhancement is meaningful, including approaches such as prompt, DPO, and SFT. Accordingly, we have conducted supplementary experiments and incorporated the results in the revised manuscript.

Weakness 1: ...However, retrieval-augmented generation is also a way of "factuality enhancement", and that certainly steers models away from parametric knowledge to varying extents...

We appreciate the reviewer’s comment and would like to clarify our framing of "Factuality Enhancement" in the context of this work. In our study, the factuality of LLMs refers specifically to their ability to utilize their internal parametric knowledge. This does not involve external retrieval mechanisms, such as those employed by retrieval-augmented generation (RAG).

We understand that the reviewer might be interpreting "Factuality" more broadly as "accuracy"—evaluating whether the model provides correct answers based on the objective world, regardless of whether the information originates from its parametric knowledge or context knowledge. If so, we emphasize that "accuracy" inherently depends not only on the factuality of LLMs but also on their context faithfulness. This highlights the key problem addressed in our work: the trade-off between enhancing parametric factuality and maintaining context faithfulness.

Therefore, RAG cannot be categorized as a method of factuality enhancement within the scope of our work, as it does not aim to enhance any specific intrinsic capability of LLMs. Instead, it is a pipeline technique designed to improve response accuracy by leveraging external context.

I would like to thank the authors for their detailed response. I'm raising the rating to 6 with concerns about whether the proposed changes are too broad/extensive as camera-ready changes: the authors agree with me that the existing scope of "factuality enhancement" does not suffice and could be expanded: they did a great quantity of experiments to demonstrate that, while the new scope, framing, experiment settings, and results might be beyond the scope of camera-ready edits. I hope the area chair could weigh in on this.

We greatly thank the reviewer Fg7q for his/her helpful and insightful comments. We provide our responses to the comments as follows.

Weakness 1: In Algorithm 1, the None checks for P_new and P_para mean that once these variables are assigned a value, the algorithm stops capturing additional tokens. This setup results in identifying only the first differing token between S_new and S_para, while ignoring any subsequent distinctions. Is this interpretation correct? If so, this approach seems limited because of ignoring the subsequent tokens. Please correct me if I make it wrong.

Thank you for your insightful feedback on our Knowledge Token Capturing algorithm, which has helped us realize that the algorithm's nuances might not have been fully conveyed.

You are correct; we only capture the first knowledge-distinguishing token, which means that once it is captured, we no longer focus on subsequent tokens. Instead, we observe the logits distribution and hidden states of this first token only.

As described in our paper, we concentrate on the distinctive elements, ignoring repetitive or non-meaningful parts. To clarify this approach, let’s consider an example. Given the question, “In which country is London located?” an expected LLM output with injected context (“London is a city in America”) might be “Answer: United States,” while the original parametric output without the injected context would be “Answer: United Kingdom.” Here, $S_{new}$ is “Answer: United States,” and $S_{para}$ is “Answer: United Kingdom” in Algorithm 1.

When capturing parametric knowledge from $S_{para} = \text{“Answer: United Kingdom.”}$, the first knowledge-distinguishing token we expect to capture is “King-.” We avoid capturing “Answer:” as it holds no factual meaning, nor do we focus on “United” since it is repetitive and does not reflect the core difference. Additionally, after capturing “King-,” we do not further consider the following tokens, such as “-dom” or the punctuation.

Capturing only the first distinctive token, without focusing on subsequent tokens, is a reasonable approach because, as widely acknowledged in existing literature, the first token plays a decisive role in sequence generation. For instance, in jailbreak studies, researchers induce LLMs to produce a non-refusal token initially, which effectively “unlocks Pandora’s box.” A clearer example emerges in the logits distribution of generated tokens. When the LLM is uncertain about which token to generate, there are often two high-probability choices—such as “States” at 0.44 and “King-” at 0.45—while the probabilities of other tokens are close to zero. This reflects an "illusion" of choice between parametric and contextual knowledge, as both tokens appear "equally competitive." However, once we select “King-” as the next token, forming a new context for subsequent iterations, we observe that the likelihood of generating “-dom” next approaches 100%. This pattern is widespread and not limited to word suffixes, reinforcing that once the initial distinguishing token is chosen, the model’s generation follows this preference with strong confidence.

This demonstrates that once the first distinctive token is generated, the subsequent tokens are generated with conviction, as if the LLM has made a definitive knowledge preference based on that initial choice. Thus, our algorithm aims to capture the first distinctive token for interpretive exploration, as it best reflects the knowledge preference—whether towards parametric knowledge or contextual knowledge.

Weakness 2: It could benefit from a forward-looking research agenda or recommendations for addressing this challenge. Practical suggestions or potential strategies for mitigating these trade-offs would strengthen the contribution and provide guidance for future work.

Thank you for your valuable suggestion on proposing future work, which aligns closely with our original intent to encourage the community to focus on the trade-off between factuality and context-faithfulness, ultimately promoting the development of more trustworthy large language models.

In the final section of our paper, we offer some forward-looking insights. We believe that a high-performing LLM should integrate both factual accuracy and context-faithfulness. Therefore, we recommend that future research efforts focus on devising strategies to effectively and controllably balance these two crucial aspects. By tackling this duality, we can move closer to developing LLMs that provide accurate information while adhering faithfully to contextual cues, thereby enhancing their reliability and practical utility.

We will continue to work in this direction, and a revised version with additional recommendations would indeed strengthen the contribution.

Your constructive input remains invaluable to us, and we appreciate your dedication to enhancing the quality of our manuscript. Once again, thank you for your insightful comments.

We fully agree with the reviewer's suggestion and have provided forward-looking recommendations for future work from two perspectives: addressing the decline in context faithfulness and leveraging internal interpretative insights.

1. Addressing the Decline in Context Faithfulness Our experiments reveal that all current types of factuality enhancement methods—whether training-based, inference-based, or prompt-based—significantly reduce context faithfulness. To address this challenge, we propose considering patching strategies to mitigate the trade-offs.

Inspired by your suggestion, we employed DeCK [1] to compare decoding outputs from prompts with context-editing instructions and initial QA prompts. This approach enhances LLMs' trust in context-editing information. We conducted experiments on several factuality-enhanced LLaMA2-7B-chat models in non-decoding stages. These experiments validate the feasibility of using such strategies to restore context faithfulness.

We found that the context faithfulness of factuality-enhanced LLMs can be effectively patched using DeCK, with improvements observed across all methods. Notably, training-based methods such as Confidence-Num w/ DeCK and Self-Eval w/ DeCK showed significant enhancements, with some cases even surpassing the native performance of LLaMA2-7B-chat.

This further validates our proposed patching strategy and highlights its effectiveness, offering valuable insights for future efforts in this direction. We believe the findings from our work are highly impactful and can inspire further research within the community.

2. Leveraging Internal Interpretative Insights

We also propose drawing inspiration from the interpretative insights to further balance context faithfulness. Below, we summarize the key findings from the interpretative analysis presented in Section 5 of our paper:

(1) Factuality enhancement methods reduce the probability of new knowledge tokens while increasing the probability of parametric knowledge tokens.

(2) These methods shift the distribution of new knowledge tokens toward lower probability ranges, indicating diminished confidence in in-context knowledge compared to the baselines.

(3) Context injection alters the distribution of knowledge representations, causing them to deviate from their original factuality.

(4) Factuality enhancement methods limit the natural dispersion of hidden states after context injection. Consequently, the output decoded from these constrained hidden states remains more aligned with the original parametric knowledge, rather than effectively incorporating new contextual knowledge as seen in the baseline.

Based on the findings above, we propose the following directions:

<1> Logits Distribution Adjustments (Findings 1 & 2): Since factuality enhancement methods amplify the probabilities of parametric knowledge tokens, making it harder for in-context knowledge tokens to surpass them, a "patching" strategy could be employed to boost the probabilities of context tokens, thereby improving context faithfulness.

<2> Hidden State Distribution Control (Findings 3 & 4): Factuality enhancement methods restrict the dispersion of hidden state distributions, which limits the model's ability to integrate new contextual knowledge. Future work could explore controllable model editing approaches to regulate this restriction, enabling broader and more flexible hidden state distributions even after factuality enhancements.

We believe these insights provide valuable guidance for addressing the trade-offs between factuality enhancement and context faithfulness in future research. Thank you for raising this important point!

We greatly thank the reviewer YFbN for his/her helpful and insightful comments. We provide our responses to the comments as follows.

Weakness 1: If I understand it correctly, Algorithm 1 assumes there is one token difference between S_new and S_para. Is there any rationale behind this assumption? Additionally, if this is true, I wonder if the findings in section 5 generalize to settings where S_new and S_para different by more than 1 token?

Thank you for your insightful feedback on our Knowledge Token Capturing algorithm, which has helped us realize that the algorithm's nuances might not have been fully conveyed.

First, we would like to clarify that we do not restrict the difference between $S_{new}$ and $S_{para}$ to only one token. Rather, $S_{new}$ and $S_{para}$ represent the LLM’s textual responses to knowledge queries. For instance, consider the question: “In which country is London located?” An expected LLM output with injected context ("London is a city in America") might be “Answer: United States,” whereas the original parametric output without injected context would be “Answer: United Kingdom.” Here, $S_{new}$ is “Answer: United States,” and $S_{para}$ is “Answer: United Kingdom.”

As described in our paper, we focus on the distinctive portions, ignoring repetitive or non-meaningful elements. For example, we avoid capturing “Answer:” as it holds no factual meaning, nor do we focus on the token “United” as it is repetitive and does not reflect the key difference. Instead, our goal is to capture tokens that hold clear factual significance—those that differentiate the newly introduced knowledge from the parametric knowledge inherent in the model. In this example, a token like “Kingdom” serves as a critical marker of distinction, effectively highlighting the divergence between contextual information and the model's existing knowledge.

Capturing the first distinctive token is essential because, as acknowledged in much existing literature, the first token plays a decisive role in the generation of a sequence. For instance, in the domain of jailbreak attempts, researchers try to induce LLMs to output a non-refusal token as the first token, which effectively “unlocks Pandora's box.” A clearer example can be observed when logging the logits distribution of generated tokens. For tokens that the LLM is uncertain about, there are often two high-probability tokens—for instance, “States” with a probability of 0.44 and “King-” with a probability of 0.45—while the probabilities of the remaining tokens are close to zero. This scenario highlights an "hallucination" of choice between parametric knowledge and contextual knowledge, as both knowledge tokens appear "equally competitive." However, if we select “King-” as the next token, forming a new context for the next generation iteration, we find that the probability of the subsequent token “-dom” approaches 100%. This pattern is quite common and is not limited to word suffixes.

This demonstrates that once the first distinctive token is generated, the subsequent tokens are generated with conviction, as if the LLM has made a definitive knowledge preference based on that initial choice. Thus, our algorithm aims to capture the first distinctive token for interpretive exploration, as it best reflects the knowledge preference—whether towards parametric knowledge or contextual knowledge.

Weakness 2: I do not find other critical flaws in the paper other than some typos. For example, a closing parenthesis is missing from equation 2. In Figure 3, the legend of the green bar should be “Factuality”. There are some other few types like repeating words across the paper but these do not hinder the understanding of the paper.

Thank you for your careful review. We will address the errors you identified and make the necessary corrections to improve our paper in the revised version.

Question 1：In algorithm 1, how did you obtain S_new and S_para? Are they both contained in the MQUAKE-STUBBORN dataset?

Yes, the MQUAKE-STUBBORN dataset naturally contains both newly injected contextual knowledge and original factual knowledge, which we use as $S_{new}$ and $S_{para}$ for capturing the corresponding tokens.

Your constructive input remains invaluable to us, and we appreciate your dedication to enhancing the quality of our manuscript. Thank you for your time and consideration.

We sincerely hope that your impression and recognition of our work will be enhanced! Thank you very much!