Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse

[Clarification 7]: Why does TRUST-ALIGN without refusal HT increase the score in the ELI5 (Table 4)?

Thank you for helping to bring this typo and others mentioned in "Typos" to our attention. The value in the text for ELI5’s $F1\_{RG}$ was wrongly calculated with Trust-Score’s value; it will be corrected in the next version. As shown in the table, there is indeed an increase in $F1\_{RG}$ for ELI5 by 0.78% for the case without refusal HT. We provide a discussion below as to why an increase is observed.

We observe that the increase in overall $F1\_{RG}$ is due to the recall for answerable questions ($R\_{ans}$, proportion of answered and answerable questions over total answerable questions) increasing more than the corresponding drop in precision ($P\_{ans}$, proportion of answered answerable questions over total answered questions), resulting in an increase in $F1\_{ans}$. This increase outweighs the drop in its $F1\_{ref}$, leading to the overall rise in $F1\_{RG}$.

Table 2: Component wise breakdown of $F1\_{RG}$

---

[Clarification 8]: How does the metric and alignment method introduced in this work generalize to real-world scenarios?

It is an interesting question, thank you for asking. In critical domains like healthcare and law, one should only rely on vetted knowledge retrieved from a vetted knowledge base to ground the answer. The answers are not admissible otherwise. We may allow the LLMs to employ the commonsense reasoning skills learned during pre-training, but those must not override what has been stated in the vetted documents. This is especially important for facts, but also the rules of reasoning should they change from context to context. For instance, the legal age of driving varies from country to country, but it is typically 18 which could be encoded in the LLM parameters. If the reasoning involves such information, the LLM must use the vetted legal age of, say 21, over its parametric knowledge that states that it is 18. This is where grounded citation ($F1\_{CG}$) comes into the picture. Again, should an LLM be unable to determine the legal age in some country to answer some question (where knowing this information is necessary, but not necessarily sufficient), it should simply refuse to do so, as it cannot ground the final answer in the retrieved documents. This is an aspect that grounded refusals ($F1\_{RG}$) seeks to measure. Using the age of 18, as encoded in the parameters, cannot be truthfully grounded and thus should not be trusted. Such scenarios are meant to be captured by our proposed metrics and mitigated by alignment.

If the reviewer feels that this point should be emphasized in the draft, we would love to do it.

[Clarification 3]: Additional FRONT results demonstrating the effectiveness of Trust-Align

We thank the reviewer for raising this point. In our revised version, we have included the complete results of FRONT. Additionally, we have attached the results of FRONT across different model sizes and families (corresponding to tables 2 and 3) for your convenience below.

Trust-Align performance relative to the additional FRONT results is on par with the existing FRONT results. Moreover, Trust-Align delivers substantial improvements in $F1\_{RG}$ across all 27 configurations and enhances $F1\_{CG}$ in 24 out of 27 configurations, further demonstrating its effectiveness in improving response groundedness and citation quality.

Table 1: Additional FRONT results

[Continued below]

Instruction: Write an accurate, engaging, and concise answer for the given question using only the provided search results (some of which might be irrelevant) and cite them properly. Use an unbiased and journalistic tone. Always cite for any factual claim. When citing several search results, use 
$
1
$

$
2
$

$
3
$
. Cite at least one document and at most three documents in each statement. If multiple documents support the statement, only cite a minimum sufficient subset of the documents. If none of the provided documents contains the answer, only respond with ‘‘I apologize, but I couldn’t find an answer to your question in the search results.’’

Question: Who was looking for a heart in the wizard of oz?

Document 
$
1
$
: {passage1}    
Document 
$
2
$
: {passage2}    
Document 
$
3
$
: {passage3}    
Document 
$
4
$
: {passage4}    
Document 
$
5
$
: {passage5}  

Answer:   

Thank you for your constructive feedback. We have carefully reviewed and addressed the points you raised, and we are happy to provide further clarification if needed. If you find that your concerns have been resolved, we would be grateful if you could consider improving your score.

[Clarification 1]: Table 2 clarification

Thank you for your feedback. Trust-Align does not aim to introduce a new state-of-the-art LLM for RAG, as such, we do not identify a single best configuration. Instead, the method demonstrates how to enhance an LLM's capabilities in three key dimensions of trustworthiness: answer correctness ($EM^{F1}\_{AC}$ score), citation groundedness ($F\_{CG}$), and refusal groundedness ($F\_{RG}$) relative to its own baseline.

Table 2 aims to highlight the effectiveness of Trust-Align in enhancing the performance of each model family, rather than comparing between models. To better convey this within-model improvement, we will adjust how the results are presented, specifically, we will highlight the best values within each family.

---

[Clarification 2]: Reliability of Trust-Score as a standalone measure

Thank you for bringing up this crucial point. We agree that the standard arithmetic average (AM) may not represent the skewed score of the sub-metrics. Thus, depending on what is important, defined by the application and use case, one may take a weighted average. As far as we understand, there’s no one-size-fits-all solution in this case, as aggregation would inevitably lead to information loss. Thus, the individual sub-metrics are independently crucial.

[Clarification 2]: Statistical testing

Thank you for bringing this up. We intentionally omitted the significance testing results from the main draft to improve table readability and we are happy to discuss it here. However, if the reviewers feel these results should be included, we will make sure to incorporate them in the next version. For your reference, we provide below the results of an independent sample t-test comparing baseline models with Trust-Aligned models.

Table 2: Results from significance testing on Trust-Score

Across all datasets, we observed that t-statistic > 0 and p < 0.001 (Table 2), indicating significant improvements for Trust-Score at a significance level of 0.01.

---

[Clarification 3]: Separation of train and test set/test-set integrity

An important aspect of any research is ensuring that the validation set and test set are well separated from the training set to maintain test-set integrity, which was a paramount consideration in our experiments. Thus, we ensure that the test set remains as disjoint as possible from the training samples. Precisely, the seed samples are sourced from the training split of three candidate data sources: ASQA, QAMPARI, and ELI5, totaling approximately 10,000 samples. Evaluation is conducted on their corresponding disjoint test split (2,948 samples) to preserve the separation between training and test data.

To assess generalizability, we also tested on the out-of-domain dataset ExpertQA, comprising expert-curated samples from 32 fields such as medicine, law, history, and engineering. This evaluates the aligned models on questions distinct from training domains (e.g., Google, Reddit, and Wikipedia). We appreciate the reviewer bringing up this question and will ensure these details are included in the updated draft to avoid any further confusion.

Thank you for your detailed and constructive feedback, we answer the questions/provide clarifications below:

[Clarification 1]: Human evaluation of Trust-Align dataset

That's a great point, here we provide observations from the human evaluation. From the Trust-Align dataset, we created a data mix containing 97 samples, with approximately 20 samples representing each error type. This was followed by human evaluation where five expert annotators rated each sample. Each response was rated on three dimensions, taking inspiration from the criteria used by Gao et al. (2023) and Liu et al. (2023): (1) Correctness, (2) Citation Recall, and (3) Citation Precision. For Correctness, given a response, a set of documents, and the question, a human evaluator assesses whether the answer is correct. The answer is labeled “correct” if it fully satisfies the information requested in the question and if the claims can be inferred from the documents; otherwise, it is marked “wrong.” For Citation Recall, given a sentence and all its cited documents, human evaluators determine whether the complete set of citations “fully supports” or “does not support” the sentence. For Citation Precision, given a sentence and one of its citations, human evaluators decide whether the citation “fully supports,” “partially supports,” or “does not support” the sentence.

Table 1: Data quality review of Trust-Align data

Table 1 demonstrates a high degree of agreement (79.90%) between human annotations and our automated response labels, with 80.41% agreement on positive responses and 79.38% agreement on negative responses.

Our findings also show a high degree of agreement (76.96%) between human annotations and TRUE NLI on the necessity (precision) and sufficiency (recall) of the citations. Additionally, Cohen’s kappa coefficient between humans and TRUE suggests a moderate level of agreement (0.55), underscoring the validity of using TRUE in our data construction pipeline.

This level of agreement is on par with prior studies conducting human evaluations which typically have agreement rates in the range of 58-80% [1-4]. We appreciate the reviewer pointing out the important discussion on evaluating the alignment data quality which we will include in the appendix of the updated draft.

---

References:

[1] Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2023). Self-Instruct: Aligning Language Models with Self-Generated Instructions. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). https://doi.org/10.18653/v1/2023.acl-long.754

[2] Gao, T., Yen, H., Yu, J., & Chen, D. (2023). Enabling Large Language Models to Generate Text with Citations. Retrieved November 18, 2024, from arXiv.org website: https://arxiv.org/abs/2305.14627

[3] Kamalloo, E., Jafari, A., Zhang, X., Thakur, N., & Lin, J. (2023). HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attribution. Retrieved November 18, 2024, from arXiv.org website: https://arxiv.org/abs/2307.16883

[4] Chia, Y. K., Cheng, L., Chan, H. P., Liu, C., Song, M., Aljunied, Sharifah Mahani, … Bing, L. (2024). M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Framework. Retrieved November 18, 2024, from arXiv.org website: https://arxiv.org/abs/2411.06176

[Clarification 4]: Importance of groundedness when parametric knowledge potentially suffices.

If a model possesses the parametric knowledge to answer an otherwise unanswerable question, is "over-responsiveness" less an issue of hallucination than, say, poor instruction-following?

This is an interesting question. There are numerous scenarios where parametric knowledge may not be preferred for generating factual responses. Prominent examples include AI-based search engines like Perplexity and You.com, which prioritize grounding their outputs in web data. These platforms utilize various LLMs as information consolidators (leveraging augmented knowledge) rather than as information generators (relying on parametric knowledge). Other RAG applications where outputs must reference provided documents include legal tasks (maintaining accurate records), healthcare (summarizing patient-doctor conversations), and finance (extracting compliance details and assessing fund performance).

While parametric knowledge can indeed enhance the performance of RAG systems, its utility is most pronounced in applications where the ground-truth answers remain consistent, regardless of whether augmented information is used. In such cases, the model’s output is not conditioned on attached documents; instead, the documents serve to improve the likelihood of producing a correct answer. However, in our work, which focuses on attributed text generation, we assume that the provided documents contain the necessary answers. If the documents are insufficient, the model is expected to generate a refusal, even if it has the correct answer encoded internally in its weights. One could, however, design a system that attributes parametric knowledge when a claim is generated without referencing the documents. This would require redefining the concept of citation groundedness, as citations would no longer correspond to specific documents.

We acknowledge the reviewer’s point that poor instruction-following is a broader issue that underpins various problems, including hallucination and over-responsiveness. Hallucination can be viewed as a specific and perhaps extreme manifestation of poor instruction-following. Another manifestation is the inappropriate use of valid parametric knowledge, or "over-responsiveness," where models generate correct answers without grounding them in the provided documents. While this is not hallucination per se, it could be mechanistically correlated and warrants further research.

In this work, we specifically address the subset problem of RAG performance, focusing on reducing RAG-specific hallucinations by improving citation groundedness and ensuring appropriate refusals. Poor instruction-following is a larger challenge, and while being a strong instruction-following model is likely a sufficient condition to reduce RAG hallucinations, the reverse is not necessarily true. If the reviewer feels this discussion should be included in the main draft, we would be happy to incorporate it.

Thank you for your constructive feedback. We have carefully addressed each of your points below. If you have any additional questions or need further clarification, we are more than willing to assist.

[Clarification 1]: ...best fine-tuned models only slightly out-performed GPT-4 ... limited practical takeaways to be found

This is an excellent question. Trust-Align does not aim to introduce a new state-of-the-art LLM for RAG. Instead, it demonstrates how to enhance a given LLM's capabilities in three key dimensions of trustworthiness: answer correctness ( $EM^{F1}\_{AC}$ score), citation groundedness ( $F\_{CG}$ ), and refusal groundedness ($F\_{RG}$). The effectiveness of Trust-Align depends on the inherent capabilities of a model i.e. to what extent they can be good for RAG. For example, on ExpertQA, Trust-Align significantly increased LLaMA-3-8b Trust-Score from 38.26% to 54.85% making it better than GPT-4 on refusal (67.07% vs 52.91%) and citation groundedness (70.11% vs 69.83%). While it is not our goal to achieve SOTA, given the trend observed: LLaMA 3b (49.0%) < LLaMA 7b (51.8%) < LLaMA 8b (54.85%), one can potentially achieve a much higher score with larger open-source models (>10b).

Observing that a (potentially) much smaller model of 8B size could outperform GPT-4 and Claude-3.5 at Trust-Score, we believe there is significant room for improvement in frontier models. These models can greatly benefit from Trust-Align to achieve a higher Trust-Score. We are eager to explore Trust-Align's potential on these models as soon as DPO support becomes available from their providers.

Thus, while choosing a non-fine-tuned frontier model can deliver performance on par with a fine-tuned sub-10B parameter model, as demonstrated across different model families such as Qwen, LLaMA, and Phi, it is advisable to fine-tune a model to achieve better alignment for RAG applications.

We appreciate the opportunity to discuss the contribution of our work and are happy to address any further questions or clarifications the reviewer may have.

---

[Clarification 2]: Clarifying Tables 2 and 3 and the corresponding comparisons presented

Thank you for your feedback. While we highlighted various findings from our experiments that may include inter-model comparisons, our main aim was to propose a method (Trust-Align) that improves a given model's RAG appropriateness as measured by Trust-Score.

Table 2 and 3 aims to demonstrate the effectiveness of Trust-Align in improving the performance of models across different families, such as LLaMA, Qwen, and Phi. The emphasis is on showcasing the improvements within each model family, rather than identifying a single model that performs best across families. To better communicate these within-family improvements, we have revised how the results are presented by specifically highlighting the best values within each model family and tuned the corresponding discussion to better convey the contributions.

---

[Clarification 3]: Confidence intervals

Thank you for bringing this up. We intentionally omitted the significance testing results from the main draft to improve table readability. However, if the reviewers feel these results should be included, we would be happy to incorporate them in the next version. For your reference, we provide below the results of an independent sample t-test comparing baseline models with Trust-Aligned models.

Table 2: Results from significance testing on TRUST SCORE

Across all datasets, we observed a t-statistic > 0 and p < 0.001 (Table 2), indicating significant improvements for TRUST SCORE at a significance level of 0.01.

[Clarification 4]: Method Clarification (Line 271)

Thanks for pointing this out. This section could indeed benefit from further clarification. The irrelevant documents used in our method are selected for their similarity to the question (cosine similarity > 0.7) but are carefully filtered to ensure they do not contain any gold claims, as determined by the TRUE NLI model. This approach ensures that we have high-quality negative examples.

From an initial pool of 100 documents pre-retrieved using GTR/BM25 (as detailed in the "Collecting D’s" section of the methods), we filter a subset of 50 documents. These documents are chosen based on their cosine similarity to the question while ensuring, via the TRUE NLI model, that they do not support any claims related to the question. From this filtered subset, we sample groups of five documents to create document sets, with questions associated with these sets labeled as unanswerable.

---

[Clarification 5]: Comparative Baseline (Line 359)

Thank you for bringing this up, we have provided an elaborate discussion on this in Appendix G.3.

Instruction-based refusal (ICL), which combines in-context examples with a refusal instruction, is a key baseline in our study. Results for this baseline are presented in Tables 2 and 3 of the main paper under rows labeled "ICL."

While Tables 2 and 3 include refusal prompt results only (ICL), we conducted a small study on frontier models (GPT-4, GPT-3.5, Claude-3.5) and Trust-Aligned LLaMA to determine how the refusal prompt compares to the default prompt. As shown in Table 12, the refusal prompt (R) outperforms the default prompt (D).

Templates:

• Default prompt: Write an accurate, engaging, and concise answer for the given question using only the provided search results…
• Refusal prompt: [Appends to the Default Prompt] If none of the provided documents contains the answer, only respond with 'I apologize, but I couldn't find an answer...’

Table 1: Refusal and default prompting results on LLaMA-2-7B (more models can be found in the paper appendix)

Taking LLaMA-2-7b as an example, models rarely refuse under the default prompt (AR% close to 100), while adding a refusal prompt in ICL drastically reduces AR%, often to near zero, indicating indiscriminate refusal. At both extremes, Trust-Score scores suffer due to errors in correctly refusing questions and lower citation groundedness scores. In contrast, Trust-Align enables models to identify and correctly answer appropriate questions, resulting in nuanced refusal ability and improvements in $F1\_{RG}$. This highlights that prompting alone is insufficient to improve the model's trustworthiness effectively. Interestingly, refusal prompting appears to yield greater benefits in more capable models, such as LLaMA-2-13b and LLaMA-3-8b.

Although these findings are detailed in the appendix, we recognize their relevance to the reviewer’s question and will incorporate this discussion into the main paper for added clarity.

---

[Clarification 6]: Clarification on SFT Model (Line 387)

Yes, we first perform Supervised Fine-Tuning (SFT) on the based model with the positive responses (r+) from the Trust-Align dataset. Performing SFT prior to Direct Preference Optimization (DPO) is part of the standard preference optimization pipeline as outlined in Rafailov, R. (2023) and Ziegler, D. M. (2019). Initialization of the reference model with a supervised fine-tuned model reduces the distribution shift between the reference policy and the true reference distribution leading to training stability as noted by several works for instance Tunstall, L. (2019). The absence of the distillation SFT step often results in models failing to learn effectively from feedback, leading to poor performance. Thus, performing SFT prior to DPO is critical to training stability and ultimate model performance.

---

References:

$1$ Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C.D., & Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. ArXiv, abs/2305.18290

$2$ Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., … Irving, G. (2019). Fine-Tuning Language Models from Human Preferences. Retrieved November 19, 2024, from arXiv.org website: https://arxiv.org/abs/1909.08593

$3$ Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Rasul, K., Belkada, Y., Huang, S., Werra, L.V., Fourrier, C., Habib, N., Sarrazin, N., Sanseviero, O., Rush, A.M., & Wolf, T. (2023). Zephyr: Direct Distillation of LM Alignment. ArXiv, abs/2310.16944

Thank you for your thoughtful comments and we appreciate the feedback you have given us. We have addressed your points below and are happy to assist in any further clarifications.

[Clarification 1]: Clarifying Trust-Score notation

Thank you for the suggestion. We will update the figure and corresponding part of our article with a clearer notation system for improved readability. Since this will require changes at various places in the draft, we will incorporate them in the next version after the discussion phase ends to minimize notation conflicts during the rebuttal.

---

[Clarification 2]: Distinction Between Parametric Knowledge and Groundedness

This is an interesting question, thank you for asking! There are numerous real-world scenarios where it is critical to distinguish between parametric knowledge and external knowledge (groundedness). Parametric knowledge refers to knowledge derived from the model’s training data and stored in the model’s weights. In contrast, external knowledge pertains to information sourced from external documents or real-time data. This distinction becomes particularly important when LLMs are used as information consolidators (primarily using augmented knowledge) rather than as information generators (relying solely on parametric knowledge). The need for groundedness is especially pressing in high-stakes applications such as legal tasks (maintaining accurate records), healthcare (summarizing patient-doctor conversations), and finance (extracting compliance details or assessing fund performance), where outputs must faithfully align with the provided documents.

Another case where such distinction is important is where the answer depends on changing real-world facts. Parametric knowledge is static after training and cannot easily reflect updates in real-world information. This limitation is significant for AI-based search engines like Perplexity and You.com, where the primary role of the LLM is to act as an information consolidator, synthesizing grounded, up-to-date web search results, rather than generating responses based on potentially outdated parametric knowledge. Groundedness enables LLMs to adaptively provide contextually relevant and current information, making them more reliable for applications requiring real-time awareness.

We appreciate the opportunity to discuss the importance of distinguishing parametric knowledge from external knowledge (groundedness) and are happy to address any further questions or clarifications the reviewer may have.

---

[Clarification 3]: Choice of Models in Figure 2 (Steps 4–6)

Trust-Align constructs a preference dataset consisting of positive (preferred) responses in Step 4 and negative (unpreferred) responses in Steps 5-6. The choice of GPT-4 for Step 4 and LLaMA-2-7B for the later steps is primarily driven by the quality and quantity of responses we aimed to generate.

For positive samples, our goal was to produce responses in the format: 'statement1 $1$

$2$ statement2 $3$ .' Given the state-of-the-art instruction-following capabilities of GPT-4, it was the natural choice over open-source models (e.g., LLaMA-2-7B) for generating responses that effectively stitch together gold claims with their corresponding attributions. Notably, a range of frontier models can perform this task; however, GPT-4's greater accessibility to us was one reason we chose it over other APIs, such as Claude-3.5.

The procedure for generating negative responses is different. We focused on two aspects when choosing a model: 1) the naturalness or quality of the response, and 2) the quantity or a sufficient number of responses to perform preference optimization. We found that LLaMA-2-7B tends to generate more diverse hallucinations, which allowed us to retrieve more negative samples and consequently have a greater number of (positive, negative) tuples for preference alignment compared to GPT-4.

While simple ICL was initially explored for generating negative responses, it was found to produce outputs of poor quality. To address this limitation, fine-tuning was carried out to align LLaMA-2-7B to generate responses in the format: 'statement1 $1$

$2$ statement2 $3$ ,' without requiring explicit ICL. This fine-tuning step significantly improved the quality of negative responses, improving coherence and clarity. Thank you for asking this question and we hope that our answer clarifies the points mentioned.