FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"

We sincerely appreciate your thoughtful comments, feedback and valuable suggestions! We address each question in detail below. Please let us know if you have any additional questions or comments and we are more than happy to discuss further.

Primary questions

P1. Rationale and justifications for inconsistent context.

We appreciate the reviewer's concerns about inconsistent contexts. Here we would like to first clarify the notion of correct answer for contextual QA. When instructed to answer the question "using only the information provided in the context." (Section 3.3), the groundtruth answer is only determined based on the context, independent of the world knowledge. While we acknowledge that the example shown in Figure 2 (middle) can be seen as adversarial (contrived), the majority of questions in the 10 contextual source datasets inherently require specific contextual information to be answerable (e.g., "Who purchased the remaining 4 packages available to broadcasters?") Therefore, different contexts naturally lead to different valid answers for the same question.

To better understand the type of questions commonly appear in the contextual datasets, we provide more examples used to construct the Inconsistent Context:

Q: Where can I get a free computer?

• [Doc1] ...At my undergrad alma mater, Wake Forest, one of the chief perks is that when you showed up for freshman orientation, the school gives you a fully loaded IBM Thinkpad and a printer...
• [Doc2]...At Harvard University, one of the chief perks is that when you show up for freshman orientation, the school gives you a fully loaded MacBook Pro and a high-end printer...

Q: What is the total cost of attendance in 2012-13?

• [Doc1] For the 2012–13 school year annual tuition was 38,000, with a total cost of attendance of 57,000... (truncated)
• [Doc2] For the 2012–13 school year annual tuition was 38,000, with a total cost of attendance of 100,000... (truncated)

Q: Who delayed a nationwide switch?

• [Doc1]...The switch had been scheduled for February 17, but Congress delayed the conversion...(truncated)
• [Doc2]...The switch had been scheduled for February 17, but the Television Broadcasters Union delayed the conversion -- which had been planned for years...(truncated)

We choose to construct the new context that is highly similar to the original context. There exists a few compelling advantages for this design choice:

• Coherence: By leveraging a SoTA LLM as our context generator, we ensure that modified sentences integrate naturally into the context while exclusively supporting the new answer. Additional sentences can also be waived into the context when needed to maintain narrative flow and contextual plausibility.

• Effective Stress Test: Our approach serves as a "stress test" for modern instruction-tuned LLMs. Since these models may have encountered the source datasets during training (as evidenced by their strong performance on original tasks in Figure 5), creating contexts that are similar to the original while supporting different answers poses challenges. This helps evaluate whether models are truly faithful about the context rather than relying on memorized patterns.

We sincerely appreciate your thoughtful comments, feedback and valuable suggestions! We address each question in detail below. Please let us know if you have any additional questions or comments and we are more than happy to discuss further.

W1. This paper primarily focuses on the impact of unfaithful knowledge in documents/contexts on language models. However, there is already a key paper [1] that established a benchmark evaluation for the counter-factuality of documents, which this paper does not cite or clarify its differences from.

Thanks for pointing out! [1] is indeed a relevant prior work that should be cited and discussed, which we have now included in Section 2 Related Work. [1] is in the same line of research as ours which aims to provide a comprehensive evaluation of LLMs in contextual settings on their effectiveness and robustness under noisy retrieved information (such as counterfactual cases). We would like to highlight a few key differences:

• Domain Diversity: FaithEval aims for comprehensive coverage across diverse domains. While [1] primarily focuses on news articles, we incorporate 10 different source datasets spanning general knowledge, medicine, science, and finance, enabling a broader assessment of model capabilities.

• Scale and Quality: While [1] provides 600 base questions with 200 additional counterfactual questions and evaluates 6 LLMs from 2022-2023, FaithEval features 4.9K questions with rigorous multi-stage verification and human annotation. Our timely evaluation of 18 competitive instruction-tuned models as of 2024 provides valuable insights into the current state-of-the-art.

• Task Scope: FaithEval introduces the Inconsistent Context task, which was not explored in [1], adding an important dimension to faithfulness evaluation by testing models' ability to handle conflicting information.

• Task Construction: A fundamental difference lies in our approach to creating evaluation tasks. While [1] relies on sampling techniques (e.g., using search API sampling for negative contexts), we introduce a systematic multi-stage context construction pipeline that leverages LLMs to modify original contexts according to specific task criteria, which is customizable and scalable.

W2. While the experimental results contain extensive performance comparisons, only experiments on “Does chain-of-thought prompting improve faithfulness?” provide some empirical conclusions that reveal the existence of faithfulness issues, lacking a deeper exploration and valuable conclusions.

We appreciate the reviewer's focus on the CoT experiments, but our study reveals several additional valuable findings regarding contextual faithfulness:

• Fine-grained decomposition of contextual faithfulness: Our results demonstrate that faithfulness challenges manifest differently across tasks - strong models struggle most with counterfactual contexts (e.g,. GPT-4o drops by 48.8%) compared to unanswerable contexts (drops by 13.6%) and inconsistent contexts (improves by 7.79%). This suggests that overriding learned knowledge poses further challenges than identifying missing or conflicting information. This fine-grained understanding of different faithfulness challenges helps identify specific areas where model improvements are needed.
• Impact of model size and family: Our extensive evaluation across 18 representative LLMs provides valuable insights about faithfulness. For each context type, our benchmark features an extensive number of models with detailed comparison for original vs. new task. We reveal that larger models do not necessarily exhibit better faithfulness (e.g., GPT-4o achieves only 47.5% on counterfactual contexts despite 96.3% accuracy without context), suggesting that current scaling approaches may not inherently solve faithfulness issues.
• Sycophancy: We found that adding explicit instructions for handling specific scenarios (like identifying inconsistencies) can actually degrade performance on normal contexts by 5% (Table 1, Claude 3.5 Sonnet), revealing a concerning tendency toward sycophantic behavior that compromises faithfulness.

While we agree that deeper analysis with advanced inference technique on contextual faithfulness is valuable, presenting the first large-scale fine-grained benchmark with multi-stage context construction and validation framework is equally valuable to the community.

Additionally, our framework provides a scalable approach for generating training data with reduced human annotation, enabling future work on supervised fine-tuning and alignment for improved contextual faithfulness.

We sincerely appreciate your thoughtful comments, feedback and valuable suggestions! We address each question in detail below. Please let us know if you have any additional questions or comments and we are more than happy to discuss further.

W1. Overlap with Prior Work: The three tasks—unanswerable, inconsistent, and counterfactual—each has been partially covered in prior benchmarks. The paper could clarify its unique contributions in methodology and purpose, with more detailed comparisons to similar works [1-4].

Thank you for highlighting these relevant works. We have thoroughly updated Section 2 (Related Work) to include these works. Below, we provide a detailed comparison:

• Scale and Comprehensiveness: While most existing works cover a few specific domains ([1] on news articles, [2] on Natural Questions and CovidNews), FaithEval offers:

  • Coverage across diverse domains with 11 source datasets (e.g., general knowledge, medicine, science, finance)
  • 4.9K high-quality questions (compared to 600 in [1])
  • Evaluation of 18 state-of-the-art instruction-tuned models as of Sep 2024 (compared to 6 in [1] and 2 in [2])
  • Rigorous multi-stage verification including human annotation

• Distinct Focus and Design Choice:

  • [1] uses sampling-based approaches (e.g., search API) for generating counterfactual contexts. In contrast, we leverage LLMs as generators to enable more flexible customization and diverse task criteria. Both approaches are valid design choices with different trade-offs.
  • [2] investigates misinformation's impact on factuality. In contrast, we focus on contextual faithfulness where the ground truth answer is determined by the provided context.
  • [3] and related works (Self-RAG [5], CRAG [6]) focus on defending against malicious retrieved passages and improving RAG under noisy retrieval. Finding solutions is orthogonal to our work, as we focus on diverse and scalable benchmark construction tailored to contextual faithfulness. Investigating how these advanced frameworks improve contextual faithfulness represents valuable future work.
  • [4] examines conflicts between parametric knowledge and external evidence, which complements our counterfactual context task. However, our work explores broader scenarios including unanswerable and inconsistent contexts.

In addition, unlike prior benchmarks [1,4], we incorporate meticulous human annotation, which we believe would be valuable to ensure that the benchmark contains high-quality contextual QA pairs. Therefore, our work complements and fills the gaps in existing works.

[1] Jiawei Chen, et al. "Benchmarking Large Language Models in Retrieval-Augmented Generation". arxiv.org/abs/2309.01431

[2] Yikang Pan, et al. "On the Risk of Misinformation Pollution with Large Language Models". arxiv.org/abs/2305.13661

[3] Chong Xiang, Tong Wu, et al. "Certifiably Robust RAG against Retrieval Corruption". arxiv.org/abs/2405.15556

[4] Kevin Wu, et al. "ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence". arxiv.org/abs/2404.10198

[5] Asai et al., Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.

[6] Yan et al., Corrective Retrieval Augmented Generation.

Thank you for the detailed response! I appreciate the clarifications provided, which have mostly addressed my concerns regarding W1, W3, and W4.

For W2, my concern lies specifically with the use of the “original context” in the unanswerable context setup. Since the original context might not be genuinely “answerable,” the evaluation on it seems less convincing to me. Could you elaborate on any strategies or measures taken to ensure that evaluations on the “original context” are reliable?

Additionally, it seems that the two questions I raised in my earlier feedback were not directly addressed.

Thank you again for your valuable feedback and acknowledgment of our responses on W1,W3, and W4. We would like to provide further details and clarifications for the remaining questions.

For W2, my concern lies specifically with the use of the “original context” in the unanswerable context setup. Since the original context might not be genuinely “answerable,” the evaluation on it seems less convincing to me. Could you elaborate on any strategies or measures taken to ensure that evaluations on the “original context” are reliable?

• The "unanswerability" set up does not use the original context but rather uses the modified context which is unanswerable as detailed in the previous response.
• The experiment on the original context is mainly used as reference to better demonstrate the performance gap and is not the focus of this work. We do not require that the original context alone is answerable without reliance on common sense/parametric knowledge. For the original contexts, we adopt the typical evaluation setups in prior works where the instruction-tuned LLM is provided with the context and a question. The model is expected to utilize both parametric knowledge and context information to answer the question.

Q1: In Section 3.2, for the counterfactual context task, it’s mentioned that a context passes if all words from the answer are present in the context. Given this criterion, why not use an automated method instead? Additionally, if the context includes a phrase like “[answer] is wrong,” would that still be considered as passing?

Indeed, we use automated method (string matching) for Counterfactual Context verification. As specified in L258-260, "we validate using a string-based matching method, where the context passes if all words from the answer appear in the context." This decision was based on our pilot human studies that showed nearly perfect agreement with the string-matching method on generated contexts based on ARC-Challenge.

We have conducted another careful review of the generated contexts, and confirm that no phrases such as '[answer] is wrong/incorrect' appear. Such negations are effectively prevented mainly due to the use of a SoTA LLM as context generator with carefully designed prompts that require coherence. A concrete example is shown in Figure 20, where the context offers both scientific observations across centuries and historical practices with early shipbuilders in Scandinavia, both of which support the counterfactual fact that wood exhibits magnetic properties.

Q2: Is the prompt used for setting up the counterfactual task specified in paper?

Yes, we use the default prompt (L274, Default Evaluation Scheme) for the counterfactual task: "You are an expert in retrieval-based question answering. Please respond with the exact answer, using only the information provided in the context. Context: {context}\n Question: {question}\n Options:{options}\n Answer: ".

We have modified the manuscript to make the above points clear. Please let us know if this has clarified your questions.

Dear Reviewer 14j3,

As we are approaching the end of the rebuttal period, we want to ensure there is enough time to address any follow-up questions you might have regarding our response or the updated paper.

Please let us know if our new responses have resolved your remaining questions. We are happy to address any last-minute questions before the rebuttal period concludes.

We sincerely appreciate your thoughtful comments and suggestions! We address each question in detail below. Please let us know if you have any additional questions or comments and we are more than happy to discuss further.

W1.1 About unanswerable questions: The paper uses prompts to control that answers must be based on context, even though LLMs might know these answers. This approach seems to test instruction following rather than true reasoning. Why not create synthetic questions that LLMs couldn't know beforehand?

In this work, we focus on contextual faithfulness instead of parametric knowledge of LLMs. As acknowledged by Reviewer 5P3K, the FaithEval benchmark "focuses on the issue of faithfulness hallucination - instances where model-generated content either deviates from the given context or includes unverified information." In contextual settings such as RAG and summarization , relying on the information presented in the context is critical.

We curate the FaithEval benchmark independent of individual models' parametric knowledge (which inherently diverges across model families) through two key principles:

• Models should answer questions when sufficient information exists in the context, even if their parametric knowledge suggests otherwise.
• Models should acknowledge uncertainty ("unknown") when the context lacks necessary information, even if they know the answer from training.

While creating synthetic questions that LLMs couldn't know is interesting, it presents several challenges:

• The rapid development of LLMs and expanding training datasets make it increasingly difficult to ensure questions will always remain unknown to future models
• Such questions might not reflect real-world scenarios where models must prioritize contextual information over their knowledge

Instead, using questions that models might know offers a more practical "stress test" of contextual faithfulness. As shown in Figure 5, the models' strong performance on original tasks suggests familiarity with the source datasets. This setting helps us evaluate whether models can remain faithful to context, and override their parametric knowledge when it conflicts with context.

W1.2 Why not analyze cases where LLMs answer correctly despite a lack of context support?

We analyze the performance of LLMs both with and without context support depending on the nature of the source datasets:

• For Counterfactual Context, we have provided an analysis of the original task (ARC-Challenge) without context in Sec 4.3 (Figure 6). ARC-Challenge is a non-contextual dataset with questions on universal scientific principles that have clear, context-independent answers.

• In contrast, Unanswerable Context is derived from contextual QA datasets, where most questions inherently require specific context to be answerable (e.g., "Who purchased the remaining 4 packages available to broadcasters?"). Testing these questions without context primarily measures model memorization, which is heavily influenced by various factors such as pre-training data, model architecture, and post-training strategies.

We conducted an additional experiment evaluating models on these questions without context. The results below show the average accuracy across 10 source datasets:

We observe that (1) the performance without context varies significantly (26.8% to 48.0%). (2) There is no strong correlation between performance with and without context. (3) Models generally perform much better with context (62.6-76.5%) than without (26.8-48.0%). However, the performance gap varies substantially by model family.

This analysis reinforces the value of our benchmark for evaluating context faithfulness, as it specifically tests whether models can remain faithful to the provided context—especially when that context suggests a different answer than what they may have memorized during training.

Dear Reviewer Ykwg,

As we are approaching the end of the rebuttal period, we want to ensure there is enough time to address any follow-up questions or concerns you might have regarding our response or the updated paper.

Please let us know if our responses have resolved your concerns. We are happy to address any last-minute questions before the rebuttal period concludes.

Thank you again for your thoughtful feedback and follow-up questions.

If the difference between the model family affects the eval results?

Indeed, we observe that the model family significantly affects the evaluation results (assuming that you are referring to the Unanswerable Task of FaithEval; other tasks share similar observations). While strong models generally perform well on original contexts, this performance does not translate to handling unanswerable contexts effectively (Command R+ ranks #2 on Original but drops to #10 on Unanswerable). As detailed in Table 7 (Appendix G), these ranking shifts reveal that different model families handle context faithfulness differently.

If so, from my perspective, should the model say no to user queries under the faith eval questions?

Regarding whether models should say no to queries in Unanswerable Task: Yes, in FaithEval, we explicitly instruct models to respond with "unknown" when information is not available in the context (L278) (task instructions are also added to the Inconsistent Task). Models can also use other phrases indicating refusal to answer (Appendix A, L875). As detailed in the response above (W1.1), this could be important for real-world applications where models must prioritize context faithfulness over their parametric knowledge, such as applications with private client data. We note that FaithEval offers flexibility in the task prompt, which can be customized under other principles, depending on the applications.

We look forward to your updated evaluation score and are happy to address any other questions you might have!

W2: When constructing Inconsistent Context, only the sentence in the original context that contains the old answer is changed to a new answer, while the rest remains unchanged. This approach makes the new context highly similar to the original context...which does not seem to be a desirable characteristic for a good dataset.

Indeed, we choose to construct the new context that is highly similar to the original context. There exists a few compelling advantages for this design choice:

• Coherence: By leveraging a SoTA LLM as our context generator, we ensure that modified sentences integrate naturally into the context while exclusively supporting the new answer. For example:

  • Question: "The oil crisis caused oil companies to increase oil supplies in which area?"
  • Original context: "...The embargo left oil companies searching for new ways to increase oil supplies, even in rugged terrain such as the Arctic. Finding oil and developing new fields usually required five to ten years before significant production."
  • New context: "...The embargo left oil companies searching for new ways to increase oil supplies, even in harsh environments such as the Sahara Desert. Finding oil and developing new fields usually required five to ten years before significant production."

  Both Sahara Desert and Arctic fits naturally into the "harsh environment". Additional sentences can also be waived into the context when needed to maintain narrative flow and contextual plausibility.

• Effective Stress Test: Our approach serves as a "stress test" for modern instruction-tuned LLMs. Since these models may have encountered the source datasets during training (as evidenced by their strong performance on original tasks in Figure 5), creating contexts that are similar to the original while supporting different answers poses challenges. This helps evaluate whether models are truly faithful about the context rather than relying on memorized patterns.

An alternative Inconsistent Context. We also acknowledge the reviewer's concern and used our framework to create an alternative Inconsistent Context where the new context is not similar to the original one. Instead of modifying the original context, we create entirely new contexts by prompting the LLM to generate multi-paragraph text that supports the new answer, similar to our Counterfactual Context construction pipeline (Section 3.1). We denote this variant as Inconsistent Context (Non-Adv). We provide a concrete example:

• Question: What financial issue is notoriously prevalent in the construction field?
• Original Context: Construction projects can suffer from preventable financial problems. Underbids happen when builders ask for too little money to complete the project. Cash flow problems exist when the present amount of funding cannot cover the current costs for labour and materials, and because they are a matter of having sufficient funds at a specific time, can arise even when the overall total is enough. Fraud is a problem in many fields, but is notoriously prevalent in the construction field. Financial planning for the project is intended to ensure that a solid plan with adequate safeguards and contingency plans are in place before the project is started and is required to ensure that the plan is properly executed over the life of the project...
• Original Answer: Fraud
• New Context: The construction field has long been known for its challenges in financial management, from ensuring accurate bids to managing cash flow effectively. Companies often engage in various forms of community outreach to enhance their public image and establish goodwill. While such activities can be beneficial, they sometimes lead to financial strain when firms make excessive charity donations, exceeding their planned budgets for corporate social responsibility. This issue has gained attention in recent years as some construction firms have faced budget overruns and delays in projects due to unanticipated charitable expenditures...
• New Answer: Excessive charity donations

We observe that Inconsistent Context (Non-Adv) still presents significant challenges, especially for open-source models. For example, Llama-3.1-8B-Instruct improved from 37.2 (Adv) to 51.4 (Non-Adv), compared to 81.5 (Original Context). The results are averaged over ten datasets. Similar trends hold for Phi3-medium-128k-instruct (4.0 vs. 9.5 vs. 81.7) and gemma-2-9b-it (27.1 vs. 36.4 vs. 83.2). Due to the time constraint of the rebuttal period, we plan to include comprehensive evaluation results for all models in the final version.

Thank you for your valuable feedback, thoughtful comments, and your support of our work! We address each question in detail below. Please let us know if you have any additional questions or comments and we are more than happy to discuss further.

W1.1: How frequent does multi-hop reasoning (multiple sentences in the context might need to be combined to infer the answer) exist in the original datasets?

Our source datasets are primarily composed of single-hop questions, where answers can be found in a single span of text. Specifically, eight of the ten datasets (SQuAD, NewsQA, TriviaQA, NaturalQuestions, SearchQA, RACE, TextbookQA, and BioASQ) are single-hop in nature. Only two datasets involve multi-hop reasoning: HotpotQA, which was explicitly designed for multi-hop questions, and DROP, which includes some questions requiring the combination of multiple pieces of information and numerical reasoning across them.

W1.2: Are there improved methods for constructing Unanswerable Context to address this issue?

We are aware of multi-hop reasoning and the current four-stage context construction framework has been designed to handle both single-hop and multi-hop reasoning cases:

• Context Construction (Fig. 17): the LLM is instructed to review the original context sentence by sentence and modify sentences that mention the old answer. We observe that in multi-hop reasoning cases, where answering requires combining multiple sentences, the groundtruth answer is often mentioned in at least one sentence. Removal of such sentences effectively breaks the reasoning chain and makes the question unanswerable given the new context. Here is a concrete example from HotpotQA:

  • Question: "Who is older, Tarryl Lynn Clark or Michele Marie Bachmann?"
  • New context: "Tarryl Lynn Clark is a Minnesota politician and a former member of the Minnesota Senate. A Democrat, she represented District 15, including portions of Benton, Sherburne, and Stearns counties, from 2006 to 2011. She was a Democratic-Farmer-Labor Party nominee for United States Congress in 2010, unsuccessfully challenging incumbent Republican Michele Bachmann... Michele Marie Bachmann is an American politician. She is a former member of the United States House of Representatives, who represented Minnesota's 6th congressional district, a post she held from 2007 to 2015. The district includes several of the northern suburbs of the Twin Cities, as well as St. Cloud."
  • Justifications provided by the context generator LLM: "The birthdates of both Tarryl Lynn Clark and Michele Marie Bachmann have been removed from the context. Without this information, it is impossible to determine who is older based on the modified passage."

• Context Verification (Fig. 3 and Sec. 3.2): To further mitigate errors in the above step, we employ two verification processes. First, during auto-evaluation, a separate LLM judge assesses each (new context, question) pair, providing a binary score with detailed justifications about the context's unanswerability. Determining unanswerability requires reasoning over the whole context. This is followed by human validation, where three Mechanical Turk workers verify the unanswerability using a majority-vote approach. This dual-verification process helps catch and filter out potential errors from earlier stages (over 98% agreement among human annotators).