ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability

Q3: While the performance of AARF seems good (Figure 6) - it would be good to see some example outputs - its unclear how this could effect the model’s output in terms of coherence/writing in general.

A3: Thank you for your suggestion. In the revised paper, we have added a new section, Appendix O: CASE STUDY FOR AARF (Line 1349), which analyzes the results of AARF on the LLaMA2-7B RAGTruth dataset. Due to the length of the examples, we only present one example here (A is the AARF and B is the Baseline):

Example Two:
Instruction: Briefly answer the following question:
how to boil potatoes for easy peeling
Bear in mind that your response should be strictly based on the following three passages:
passage 1:Boil until tender, then transfer to an ice water bath. When cool, peel off skin. (15 minutes to cook, 5 seconds to peel each cooked potato) And now, an epic demonstration of How to Peel 2 Pounds of Potatoes in Under A Minute using a drill, (clean) toilet brush, bucket, and water hose. GAME. CHANGER.

passage 2:This versatile medium-starch spud has a delicious, buttery flavor. It cooks up tender and holds its shape, but it also has just enough starch to contribute creaminess without making a potato salad mushy. Its skin can be tough and papery, so we recommend peeling them first.

passage 3:Especially if you are peeling a crap load of potatoes. Here is the easiest way (or at least the easiest way I’ve discovered) to peel a cooked potato. Step 1.) Bring a pot of water to a boil. Step 2.) With a knife, score your potatoes all the way around the center. Step 3.) Place your potatoes in the pot and boil for about 15 minutes or so. Step 4.) Prepare a bowl of ice water and place your potatoes in the water. Step 5.) Swish it around for a few seconds just enough so that it’s cool enough to handle. Step 6.) Remove your potatoe from the water and remove the skin.


A' Response: Based on the provided passages, the best way to boil potatoes for easy peeling is to:

1. Boil them until tender, then transfer them to an ice water bath to cool down quickly (as shown in passage 1).
2. Peel the potatoes as soon as they are cooked, as the skin can become tough and papery if left to sit (as stated in passage 2).
3. Boiling them, scoring the skin, cooling them in ice water, and then effortlessly removing the skin. (as demonstrated in passage 3).


B's Response: Based on the provided passages, the best way to boil potatoes for easy peeling is to:

1. Boil them until tender, then transfer them to an ice water bath to cool down quickly (as shown in passage 1).
2. Peel the potatoes as soon as they are cooked, as the skin can become tough and papery if left to sit (as mentioned in passage 2).
3. Use a drill, toilet brush, bucket, and water hose to peel 2 pounds of potatoes in under a minute (as demonstrated in passage 1).


GPT4-o Judge Result: A's answer accurately summarizes and organizes the critical steps for boiling potatoes for easy peeling, referencing the instructions from the passages sequentially. It correctly includes the process of boiling, cooling in an ice water bath, and peeling. 

From the results of the examples presented in Appendix O, we observe that AARF enables the model to:

• Generate more comprehensive responses (Case One),
• Attribute cited content more accurately to the correct passage (Case Two), and
• Reduce fabricated hallucinations (Case Three),

without negatively affecting coherence or writing quality. These findings demonstrate the advantages of AARF in reducing hallucinations in RAG scenarios.

Q1: Performing this analysis at the token/chunk level might limit its practicality in real time or large scale settings - it would be nice to have a richer discussion of the trade-offs and real world feasibility.

A1: We appreciate the your concern about the practicality of token/chunk-level analysis in real-time or large-scale applications. However, ReDeEP is designed to calculate PKS and ECS metrics in parallel during response generation, ensuring low latency. As detailed in Appendix L Line 1232, our efficiency analysis shows that ReDeEP achieves competitive performance among hallucination detection methods, validating its industrial feasibility. While ReDeEP achieves high efficiency and low latency, we acknowledge that additional optimizations, such as selective chunk-level analysis or adaptive granularity, could further enhance its applicability in extremely resource-constrained or large-scale scenarios.

Q2: The experiments are extensive - however they are all with the LLama family of models - testing (even a much smaller set) on a different model would be informative.

A2: Thank you for your question. To address your concern, we have added experiments on the Mistral-7B-Instruct model using the RAGTruth dataset. Due to time constraints, we selected representative baselines and compared them against ReDeEP (Chunk). The results demonstrate that ReDeEP (Chunk) outperforms baseline models on most metrics, showing that ReDeEP can generalize beyond the LLama family to effectively detect hallucinations.

These results highlight the capability of ReDeEP to detect hallucinations effectively on models outside of the LLama family, thereby validating its generalizability.

Q1: Figure 3 is problematic.

A1: Thank you very much for your insightful suggestions. We have carefully revised Figure 3 based on your feedback. Specifically:11_

1. To reduce ambiguity and improve clarity, we removed non-essential components such as the FFN and attention components for tokens $t_1$ and $t_2$, retaining only the main residual stream.
2. To simplify the diagram and reduce visual clutter, we removed token $t_3$, which also reduced the number of arrows.
3. We added more detailed descriptions for input and output variables, isolating the computation graphs for the Parametric Knowledge Score and External Context Score to better define their computation processes.
4. Panels (b) and (c) have been removed to focus solely on the main model diagram.
5. The font size has been increased to enhance readability.

The updated version of Figure 3 can be found in our revised paper.

Q2: The number of hallucinations and non-hallucinations (h = 0 and h = 1) in the evaluation set

A2: Thank you for your question. The number of hallucinations (h = 1) and non-hallucinations (h = 0) in the evaluation set are summarized below:

Q3: Question about Pearson's Correlation in §3.

A3: Thank you for your question. The purpose of using Pearson's correlation in Section 3 is to study whether there exists a statistically significant linear relationship between the External Context Score (ECS)/Parametric Knowledge Score(PKS) and hallucination labels. This analysis is aimed at validating our hypothesis about the relationship between the two variables, rather than assessing detection performance. While detection metrics such as AUROC and accuracy provide insight into the classification performance, Pearson's correlation serves as a measure of the statistical relationship between ECS/PKS and hallucination labels. In the experimental section, we have already included AUROC and other detection metrics to assess the classification performance of using ECS and PKS in detecting hallucinations. The use of Pearson's correlation in Section 3 is limited to an exploratory analysis to identify potential linear trends between the variables, rather than as an evaluation metric for the detection task.

Q4: Criteria for selecting thresholds for accuracy, recall, and F1

A4: Thank you for your question. We have summarized the criteria for selecting thresholds for accuracy, recall, and F1 in the following table:

Q5: Use full names for FFN, ReDeEP, and AARF, at least in the abstract.

A5: Thank you for your question. We have updated the abstract to replace "FFN" with its full name. Additionally, we have included explanatory modifiers for "ReDeEP" and "AARF" in the abstract to ensure better clarity and understanding for readers.

Q6: In Figure 4(c), clarify what the colour bar values represent.

A6: Thank you for your suggestion. In Figure 4(c), the color bar values represent the copying head scores. We have updated the title of Figure 4(c) in the revised paper to clarify this.

Q7: Structure in §3.2 is difficult to follow. Stick to a standard structure using \section, \subsection, \subsubsection, \paragraph, etc.

A7: Thank you very much for your suggestion. The structure in §3.2 was organized this way primarily to save space, considering the strict page limitations for the main body of ICLR submissions. We will adopt a standard structure in other versions of the paper to make it easier for readers to follow.

Reference:

[1] Kortukov, E., Rubinstein, A., Nguyen, E., & Oh, S.J. (2024). Studying Large Language Model Behaviors Under Context-Memory Conflicts With Real Documents.

[2] Transformer Circuits Thread, 2021. URL https://transformer-circuits.pub/2021/framework/index.html.

[3] Wu, Y., Zhu, J., Xu, S., Shum, K., Niu, C., Zhong, R., Song, J., & Zhang, T. (2023). RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models. Annual Meeting of the Association for Computational Linguistics.

[4] Hu, X., Ru, D., Qiu, L., Guo, Q., Zhang, T., Xu, Y., Luo, Y., Liu, P., Zhang, Y., & Zhang, Z. (2024). RefChecker: Reference-based Fine-grained Hallucination Checker and Benchmark for Large Language Models. ArXiv, abs/2405.14486.

[5] Jin, Jiajie, et al. "FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research." arXiv preprint arXiv:2405.13576 (2024).

[6] Chen, Chao, et al. "INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection." The Twelfth International Conference on Learning Representations.

[7] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024. LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics.

Q10: Anonymous prompt for Hallucination generation experiment.

A10: Thank you for your suggestion. Following your advice, we conducted the evaluation using anonymous names to mitigate potential naming bias. We ran the evaluation twice: in the first run, AARF and Baseline were anonymized as A and B, respectively; in the second run, their roles were reversed, with AARF and Baseline anonymized as B and A. The final results were obtained by averaging the outcomes of these two evaluations:

We observed that compared to the results without anonymization, the number of ties determined by the LLM increased, and the proportion of AARF wins slightly decreased. However, AARF still consistently outperformed the Baseline. This experiment demonstrates that the robustness of AARF's performance persists even under conditions designed to eliminate naming and positional biases.

Q11: Lack of sensitivity experiments

A11: Thank you for your suggestion. We conducted sensitivity analysis experiments on both ReDeEP(Token) and ReDeEP(Chunk) based on the parameter ranges provided in Section J: IMPLEMENTATION DETAILS. The experimental results have been added to the revised paper in Appendix N: SENSITIVITY ANALYSIS.

From the results, we observe a trend of performance either increasing and then stabilizing or decreasing slightly. The overall impact of any single hyperparameter on the model's performance is minor, reflecting the robustness of our approach.

Additionally, we note that the sensitivity curves for the same hyperparameter are similar across different models. For most hyperparameters, the performance extremes appear within the same parameter ranges across models, further demonstrating the cross-model stability of ReDeEP.

Please refer to the revised paper Appendix N: SENSITIVITY ANALYSIS (Line 1337--1347) for detailed results.

Q12: Unclear writing: Question about the Copying Heads (set A) Knowledge FFNs (set F)

A12: Thank you for your question. Your understanding is correct. We have clarified the details in our revised paper.

Q13:Strange ordering equations.

A13: We sincerely apologize for the confusion caused by the ordering of equations. Due to the page limit of the main text, we moved the introduction of $a$, which refers to attention weights, to the appendix. This term represents a commonly used concept in the context of our formula, which is why we chose to provide its detailed explanation in Appendix Eq. 8.

Q14:Why LLama2-7B (smaller and older version than others) has better results on Dolly in terms of F1 or Accuracy in Table 1?

A14: Thank you for your question. The RAGTruth and Dolly (AC) datasets we used are annotated based on the outputs of different models. Since LLaMA2-7B and LLaMA2-13B may produce different outputs for the same question, the corresponding annotated data also differs. As a result, the metrics are not directly comparable between models. It is possible for a smaller model to outperform a larger model on certain metrics.

This phenomenon is also observed in other hallucination detection papers, such as [6], where in Table 1, LLaMA-7B outperforms LLaMA-13B on the CoQA dataset.

Q5: Same Baseline comes for Dolly dataset, please include results for AlignScore and RepC-LE-nn-n2000-e2000

A5: The Dolly dataset originates from the RefChecker model paper[4]. AlignScore and RepC-LE-nn-n2000-e2000 are components of RefChecker, and the "84 and 86 accuracy" mentioned for these methods are derived from Table 8 in RefChecker [4]. However, the experiments in Table 8 use 11k human-annotated claim triplets, and while the column is labeled as "Accurate-context," the dataset and task are entirely different from the hallucination detection task in our paper.

For hallucination detection, the relevant results are presented in Table 2 of RefChecker. In our experiments, we compared with RefChecker and ensured fairness by setting both the extractor and checker to gpt4-o-mini as described in Appendix I. From Table 1 in our paper, it can be observed that our model outperforms RefChecker, demonstrating the effectiveness of our approach for the hallucination detection task.

Q6: Please also provide results for the Noisy Context split from Dolly [4] dataset.

A6: Thank you for your suggestion. The Noisy Context split from the Dolly [4] dataset may not be suitable for our scenario. As stated in our paper Line 77-78, our work focuses on detecting RAG hallucinations specifically in cases where the retrieved external context is accurate and relevant.

Our method is designed for scenarios where users upload personal knowledge bases or in realistic RAG application scenarios where filters are applied to remove misinformation and irrelevant information. Such filters are well-studied in fields like misinformation detection and information retrieval, which can largely ensure that the context provided to the RAG model is accurate and relevant.

Q7: Causal experiment: question about the NLL difference.

A7: We apologize for any confusion caused. Our intervention experiments were conducted on the RAGTruth dataset, which contains truthful answers. The NLL difference is used to measure the change in the negative log-likelihood of the model generating truthful answers before and after the intervention. If the experimental group shows a larger NLL difference compared to the control group, it indicates that the intervention in the experimental group more significantly disrupts the model's ability to produce truthful answers (i.e., leads to hallucinated answers).

As shown in Figure 5 (Left), the NLL difference is greater for interventions on Copying Heads (experimental group) than for non-Copying Heads (control group). This demonstrates that Copying Heads are more strongly associated with RAG hallucinations compared to non-Copying Heads.

We have added a more detailed explanation of this to the revised paper in Appendix E: DETAILED INTERVENTION PROCEDURES.

Q8: Causal experiment: question about the noisy results in Figure 5 (b).

A8: Thank you for your question. We acknowledge that Figure 5 (Left) (b), exhibits significant noise and does not effectively demonstrate the impact of FFN interventions. We agree that the intervention experiment you proposed in the "No PKS justification" section better illustrates the causal relationship between FFN layers and RAG hallucinations. Therefore, in the revised version of our paper, we have incorporated the parametric knowledge bias experiment into Section 3.2 EXPERIMENTS (RQ2) to provide stronger evidence that the Knowledge FFNs within LLMs excessively injecting parametric knowledge into the residual stream can lead to RAG hallucinations.

Q9: How many heads are within experimental and control groups.

A9: In our original paper, we have addressed this concern in detail in the Causal Matching section of Appendix E Detailed Intervention Procedures. To ensure a fair comparison, we matched the top 32 Copying Heads with the nearest non-experimental heads within the same layer. Similarly, we matched the top 5 Knowledge FFNs, identified as being most related to hallucinations, with the nearest FFN modules in adjacent layers.

This Causal Matching approach mitigates the potential bias you mentioned, ensuring that differences in loss changes are not simply due to the experimental group intervening in more heads.

Q4: Hallucination detection experiment on Baseline LLama2-13B of RAGTruth Paper[3]

A4: Thank you for your question. The experimental setup of RAGTruth differs slightly from ours. We focus on hallucination detection for responses from individual models, whereas the RAGTruth paper mixes data from all models for evaluation. Among the baselines provided in the RAGTruth paper, we have already included all except for the fine-tuning of LLaMA2-13B.

However, the full fine-tuning of LLaMA2-13B in the RAGTruth paper was conducted using 4×A100 GPUs, which exceeds the computational resources we have available. Our 4×V100 setup cannot support flash-attention optimizations, resulting in out-of-memory issues during full fine-tuning. We also reached out to the authors of the RAGTruth paper to request testing logs to replicate on our settings, but we did not receive a response. As a result, we conducted LoRA-based fine-tuning instead, whose setting is based on [6]. The results are summarized in the tables below:

The LoRA fine-tuning results could not outperform our model. We acknowledge that full fine-tuning of large-scale LLMs on RAGTruth may yield results comparable to or better than ours. However, full fine-tuning is extremely resource-intensive and tends to overfit the current data distribution, limiting generalization capabilities. This is evident from the weaker performance of the LLaMA2-13B model LoRA-based fine-tuned on RAGTruth when tested on Dolly (AC). In contrast, our model requires no fine-tuning at all.

In deployment scenarios, our model also demonstrates higher efficiency during inference compared to LLM-based methods, further underscoring its practical potential for real-world applications.

Q1: No PKS justification: check whether it removes the parametric knowledge bias using some of the setups for that, for example, the one from [1].

A1: Thank you very much for your valuable feedback. Based on your suggestion, we followed the same experimental setup as Table 4 in [1] and used the Llama2 7B model to verify whether removing FFN layers reduces parametric knowledge bias. In our experiments, we removed the Last (32nd) Layer FFN, the 16th Layer FFN, and the 1st Layer FFN, respectively, and observed the parametric knowledge bias as defined in [1]. The experimental results are summarized in the table below:

In this table, larger values indicate greater parametric knowledge bias. The “/” symbol represents a divide-by-zero error encountered during the calculation process. Considering time constraints, we sampled the first 1,000 examples from the RAG datasets, which is a common setting in RAG scenarios [5].

The experimental results show that removing FFN layers, whether from lower layers or middle-to-upper layers, consistently reduces the parametric knowledge bias. Notably, removing middle-to-upper layer FFNs is more effective in reducing the bias, aligning with our paper’s conclusion that Knowledge FFNs are primarily concentrated in middle-to-upper layers. On the HotpotQA dataset, removing the 16th Layer FFN even resulted in a negative parametric knowledge bias. This further corroborates our conclusion that, in RAG scenarios with accurate and relevant external knowledge, excessive injection of parametric knowledge into the residual stream by Knowledge FFNs within LLMs can lead to hallucinations.

We have added this experiment to our revised paper. Please see Appendix D: JUSTIFICATION FOR PARAMETRIC KNOWLEDGE SCORE (Lines 957–980) in the revised version of our paper.

Q2: Questionable ECS justification: 1. Purpose of the experiment on Appendix C: DIVE INTO THE RATIONALE BEHIND THE EXTERNAL CONTEXT SCORE; 2. Question about whether ratio of attending is computed correctly.

A2: First, to address your first question, the purpose of this experiment was not to justify the External Context Score (ECS). For the justification of ECS, we referenced the copying heads theory proposed by Anthropic [2] (as discussed in Lines 297–307 of our paper). Instead, the goal of this experiment was to analyze the reasons behind a low ECS. In the findings presented in Lines 357–362, we identified two potential causes for a low ECS:

1. The Copying Heads may occasionally neglect necessary knowledge from the external context.
2. The LLM may lose the information retrieved by the Copying Heads during the generation process.

Regarding your second question, we greatly appreciate your suggestion. We recognized the limitations of the previous experiment and added a new experiment with a modified prompt to let GPT-4-o evaluate whether the attended span includes information that should be part of the truthful answer.

Prompt: {external context + query}

Truthful Respond: {Truthful Respond}

Given the following context information: {Attend Span}, check if it includes relevant information found in the truthful answer.

Please answer with "Yes" or "No" and provide the reason on a new line.

For validation, we used data from RAGTruth where LLaMA2-7B Chat exhibited hallucinations. We extracted truthful answers from other model responses in the RAGTruth dataset for the same questions, filtering out cases where no truthful answer existed and where the PKS was above the average value for the dataset, to minimize the impact of PKS.

As shown in the table, the results indicate that in most cases where hallucinations occur, the attention heads correctly attend to the appropriate external context. This demonstrates that a low ECS is mostly due to the LLM losing the information attended by the attention heads during the generation process.

We have added this experiment to our revised paper. Please see Appendix C: DIVE INTO THE RATIONALE BEHIND THE EXTERNAL CONTEXT SCORE (Lines 932–956) in the revised version of our paper.

Thank you very much for such a detailed response, all my questions are answered! I will increase my score to 8.

I apologize for the late response.