Studying Large Language Model Behaviors Under Context-Memory Conflicts With Real Documents

We thank the reviewer for constructive feedback and questions. We answer them below:

1.using in-context demonstration to elicit answers. However, could this introduce demonstration bias?

We understand demonstration bias for generative QA as “shown examples do not suffice to learn the task” (analogous to [a] for classification). Due to the limited context window, it could explain marginal improvement in the performance and ICL failing to prevent parametric bias (App. E). We’ll add this discussion to the revision. If we misunderstood, we kindly ask the reviewer to be more specific.

[a] https://arxiv.org/pdf/2312.07476

2.What is the accuracy of BEM? Can it ensure matching quality?

[b] showed BEM to be a stronger answer matching metric than EM. In our experiments it also outperforms EM (89% vs. 55% on original [b] data, 92% vs. 22% on our manually labeled test data from HotpotQA). We’ll add this to App. D.

[b] https://arxiv.org/abs/2202.07654

3.it's essential to include at least one closed-source LLM.

We will add results for GPT-3.5 (below).

Table 3, Conflicted open-book QA

Table 4 Diff. in the likelihood of update failure

4. Does it mean the authors have fine-tuned the models, or have they simply selected one of the best prompts?

We selected the best prompt. We did not fine-tune the models. We will clarify this in the revision.

5. Xie et al. found that LLMs are highly likely to update their parametric memory as long as the external evidence is coherent. Could you please justify the difference?

Xie et al. study coherent yet false contexts. In our setup we check behaviors for real (both coherent and factually true) contexts, which allowed us to find the parametric bias.

Q1. Could the phenomenon of 'masking reducing the likelihood of retaining parametric answers' be attributed to low-quality masking?

In our understanding, you suggest masking makes inputs OOD (“missingness bias” [c]). To minimize it, we test various strategies: using attention mask [c], varying mask tokens, using paraphrases, and changing mask size. We use the strategy with the best open-book accuracy. We ask the reviewer to elaborate if we misunderstood.

[c] https://arxiv.org/abs/2204.08945

We thank the reviewer for recognizing the merits of our work and providing constructive feedback. We address concerns in the following:

While the setup differs from previous works, the main finding seems not very novel.

Xie et al. report a similar finding for coherent yet false contexts. In our setup, we check behaviors in case of real (coherent and factually true) contexts, allowing us to find the parametric bias.

The paper does not provide a solution. It mainly focuses on analyzing and demonstrating a phenomenon.

Indeed, we discover and examine the problem and its scope, and invite the community to work together on solving it. To facilitate, we try simple solutions:

• ICL task adaptation (App. E)
• Increasing model size (App. F)
• Fine-tuning - our best result (below, will be added to revision).

We fine-tune Llama7B on the train split of each dataset.

(R) subset becomes smaller, but the model is still susceptible to adding the parametric answer to context.

Open-book QA under knowledge conflict (cf. Section 4.2)

After adding the incorrect parametric answer (cf. section 4.4.2):

in reality, the retrieved knowledge might be noisy and not necessarily be the ground truth.

Noisy retrieved documents are indeed realistic. However, prior work studied synthetic documents designed to contradict model knowledge, which is unrealistic. We study real documents, which is more realistic.

Why the happening of knowledge updates is determined by exact match?

1. To be comparable to previous work
2. RAG is designed to update model knowledge. If context had 0 influence on the answer, it is a critical failure of RAG.

Will changing it into semantic equality change the results much?

We re-ran experiments on Llama7B using BEM to determine knowledge update.

Table 3, open-book QA on the KC subset

The general trends are the same, but more examples are in R as more answer pairs are equivalent according to BEM. Please let us know if there are any further questions.

We thank the reviewer for the detailed feedback. In the following, we address the concerns:

I'm not sure settings in previous works [...] are "unrealistic". Retrieval could bring noisy, outdated, or irrelevant documents into LLM prompts

Noisy documents are indeed realistic. However, prior work studied synthetic documents designed to contradict model knowledge, which is unrealistic. We study real documents, which is more realistic.

If the setting in this work is much more prevalent, it would strengthen the motivation of this work.

RAG is often applied to trusted documents in OOD domains for the LM ​​[a,b,c]. Moreover, as retrieval improves, real-world systems will move closer to our setting. [a] https://link.springer.com/chapter/10.1007/978-3-031-60615-1_9 [b] https://arxiv.org/abs/2309.02233 [c] https://arxiv.org/abs/2403.16295

there probably is very little to investigate about knowledge conflicts in these scenarios.

The parametric bias finding shows that LLMs may read conflicting contexts incorrectly. This risk is even larger for more realistic systems (App. G).

Much of the analysis [...] focus on the 𝑅 cases [...] However, it only constitute a very small fraction of cases (<5% usually). I wonder if more effort should be spent on investigating the 𝑈𝑖 cases…

Apart from frequency, severity of a problem also matters. Answer retention (𝑅) is a severe failure of RAG. RAG is used to update model knowledge. In 𝑅, context documents did not influence the model’s answer. By analyzing these errors, we discovered the parametric bias and understood answer retention better. We'll also add an analysis of the 𝑈𝑖 cases to the revision.

we often don't know right and wrong at inference time for either parametric answers or retrieved documents.

We propose evaluating RAG beforehand on known data to check for parametric bias in Sec. 6.

The findings and proposed "interventions" rely heavily on the assumption that "parametric is wrong but retrieved is right",

We assume this to create a clean experimental setting. “Interventions” are experiments to study the phenomenon, not practical solutions.

It would be nice to present some qualitative analysis and/or examples in the main paper.

Please consider:

• Table 1: example of setup differences
• Section 4.3.1: example of parametric answer in context
• Section 4.4.2: example of adding parametric answer to the input.

We ask the reviewer to elaborate on what other examples would strengthen our work.

We thank the reviewer for the thorough review and answer the questions below.

why did you focus on the smaller fraction of answer retention errors [...]?

Frequency isn’t the only metric for problem importance, severity also matters. E.g, adversarial attacks are rare in real world. Yet, we study them. Answer retention is a severe failure of RAG. RAG is used to update model knowledge [a]. In these errors, context documents did not influence the model’s answer. By analyzing them, we discovered the parametric bias and understood answer retention better. We'll add this to the revision. [a] https://arxiv.org/abs/2005.11401

1. Table 3:[...] what is in the error category 𝑈𝑖?

We classify 100 samples in 𝑈𝑖 for the Mixtral on NQ:

• 42 incorrect answer format
• 10 model answer is also correct, but != GT
• 13 model answer is correct while the GT is incorrect
• 12 model being bad at reading tables
• 13 incorrect reading (prediction appears in context, but != GT)
• 4 answers not from context
• 5 model wrongly claims the answer is not in the context
• 3 context is illegible

We’ll add this to the appendix.

2. Table 4: Can you separate out 𝑃(𝑅|𝑎𝑝⊆𝑐)? Why did you choose to combine 𝑃(𝑅∪𝑈𝑖|𝑎𝑝⊆𝑐)?

𝑅∪𝑈𝑖 includes all wrong open-book answers. We emphasize that 𝑎𝑝⊆𝑐 biases models towards wrong answers. We agree reporting only 𝑃(𝑅|𝑎𝑝⊆𝑐) would be clearer since we only analyze this type of error. We’ll change the presentation in the revision.

For Llama7B, separating out 𝑃(𝑅|𝑎𝑝⊆𝑐) shows a stat. significant difference in answer retention likelihood for all datasets (+8.8*, +4.1*, +4.8*, +8.7*, +6.6*, +10.7*), emphasizing the parametric bias phenomenon

3. Sec 4.4.1: [...] Is the model predicting the masked (edited) parametric answer instead [...]? What do you see is the error case?

≈50% are predictions of edited answers. Others: wrong format, stated inability to answer using context, non-GT correct answers, synonyms (if also in context).

4.Sec 4.4.1: How do you draw this conclusion?

We apologize for the confusion: In masking experiments, some cases of answer retention remain. We suspect that models resort to the original answer not only due to the parametric bias but in some cases, due to limited reading abilities -- if parametric bias were the only reason, masking the parametric answer should eradicate answer retention.

5. Sec 4.4.2: What explains the increase in 𝑃(𝑈𝑖)?

The majority of these cases are due to wrong answer format.

After reading the author responses to all reviews, I am somewhat convinced by their answer to my concern.

| this category of errors contributes to a larger fraction of errors overall

The authors address this with two observations:

1. The category of errors where the model updates to a different incorrect answer is also a fragmented category. Explaining any such update would also require explanations for relatively small fractions of errors
2. They report a semantic matching metric (instead of EM) which further highlights that the "retention" errors are higher than initially estimated.

| the fraction of errors of this kind (new incorrect answer) seems to be increasing but are largely ignored

I am convinced by their breakdown of this error category.

I am increasing my score from 6 -> 7.