Understanding Parametric and Contextual Knowledge Reconciliation within Large Language Models

Dear Reviewer oNGe,

We sincerely thank you for your constructive and insightful review. Your positive acknowledgment of our novel insights for interpreting conflicting knowledge processing, carefully designed probing framework, and refined understanding of LLM internal processes is particularly encouraging and validates our research approach. We address your questions and concerns as follows:

W1. While the entity-aware probing framework avoids modifying base model weights, it introduces LoRA adapters and injects special tokens ([START], [END]) into the input. These interventions may perturb the native inference pathways. Consequently, the behavior traced by the probe may deviate subtly from that of the vanilla LLM, raising concerns about the fidelity of the interpretability conclusions.

The probe test results demonstrate high correlation with the model's actual output behavior. Additional analysis based on the LLaMA 3.1 model is provided in Appendix F of our paper, with results summarized in Table 1 below. The methods "Duplicate and append," "Insert prompt randomly," and "Tag and append" can be viewed as three approaches to enhance the model's attention to counterfactual knowledge in the context (detailed descriptions are available in Appendix F).

Table 1: Model Response Patterns Under Different Context Conditions

As shown in Table 1, when the context contains only counterfactual documents, 54.6% of responses follow counterfactual knowledge, significantly higher than responses following factual knowledge (22.7%). This phenomenon aligns with Figure 4(c) in our paper, where the information intensity of counterfactual knowledge in the output layer exceeds that of factual knowledge, and the magnitude of information intensity determines the degree of influence that response knowledge has on the output.

When we enhance the model's attention to counterfactual knowledge in the context through three different methods, the model's output does not shift from counterfactual to factual knowledge. This is consistent with our probing conclusion that "knowledge competition does not occur across categories"—that is, enhancing attention to contextual knowledge does not affect the routing of memory heads to parametric knowledge.

When the input contains both factual and counterfactual documents ($\mathcal{D}\_f\cup \mathcal{D}\_{cf}$ setting), compared to the $\mathcal{D}\_f$ setting, the model's output clearly begins to favor factual knowledge (56.4% vs. 27.7%). Why does the model favor factual knowledge when both types appear simultaneously in the context? Our probing conclusion of "superpositional reconciliation" explains this well. Factual knowledge from the context and factual knowledge from parameters form a superposition, making its information intensity exceed that of counterfactual knowledge sourced only from the context. Since information intensity magnitude directly affects output, the model favors factual knowledge.

Subsequently, when we enhance the model's attention to counterfactual knowledge, the proportion of counterfactual responses increases significantly. The results of this operation under $\mathcal{D}\_f\cup \mathcal{D}\_{cf}$ setting are strikingly different from those under $\mathcal{D}\_f$ setting, but this is not surprising. Under $\mathcal{D}\_f\cup \mathcal{D}\_{cf}$ setting, factual knowledge comes from both model parameters and context. When we enhance the model's attention to counterfactual context, the two compete for the attention of context heads, weakening the context heads' attention to factual knowledge. This aligns with our probing conclusion that "knowledge competition occurs within categories."

We understand the reviewers' concerns. Developing better probing techniques to accurately capture the internal knowledge processing mechanisms of models remains an important and open research question that merits continued exploration and improvement.

W2. A key claim is that conflicting knowledge persists in residual streams and is superpositionally reconciled rather than being suppressed. However, this assertion is not sufficiently substantiated with targeted experiments. For example, the authors do not investigate whether zeroing out high-information-intensity attention heads would alter the final model prediction.

Following the reviewer's suggestion, we provide experimental results for zeroing out context heads and memory heads in Table 2. Clearing the top-10 context heads and memory heads with the highest information intensity causally affects the model's output. This experimental finding, together with the correlation analysis provided in our response to W1, sufficiently demonstrates the interpretability of information intensity for model output behavior.

Table 2:The causal impact of context heads and memory heads on model output behavior

The response addresses the reviewer's concern about insufficient experimental substantiation by providing targeted ablation experiments that demonstrate the causal impact of high-information-intensity attention heads on model predictions, thereby supporting the claim about superpositional reconciliation of conflicting knowledge in residual streams.

Q1. Have the authors considered how toxic contextual knowledge that might trigger LLMs' safety mechanisms affects internal knowledge reconciliation? Would similar patterns of attention routing still hold?

Thank you for raising this important point. We hypothesize that toxic contextual knowledge may exhibit distinct internal processing patterns. RLHF alignment training could potentially suppress contextual heads' attention to such harmful knowledge. This represents a critical direction for future research with significant implications for RAG system robustness and alignment. While our current findings are limited to factual knowledge conflicts, they provide a foundational methodology that can be extended to investigate safety-knowledge interactions—an important area we hope to explore in subsequent work.

Q2. To what extent do the [START] and [END] tokens introduced for probing influence the internal computation of LLMs? Specifically, how much attention do LLMs allocate to these OOD tokens?

Based on the responses to W1 and W2, we can see that the probe results align well with the model's actual output behavior. We believe the influence of [START] and [END] tokens, as well as LoRA updates, on LLM working modes can be understood as follows. Under normal operating conditions, LLMs respond to user instructions by continuously predicting the next token. At each time step t, their computation is biased toward the token currently being predicted. Obviously, not every target token at each step is knowledge-intensive. We believe that [START] and [END] tokens may serve as indicators for the model to simulate the computations executed when predicting tokens corresponding to the target entity.

Dear Reviewer 6M6N,

We are grateful for your thorough review and positive assessment. Your recognition of the novelty of our approach, the significant value of our research topic, and the rigor of our experiment designs along with the depth of our conclusions motivates us and confirms the value of our research contributions. We address your questions and concerns as follows:

W1. I am still less convinced on the triple being appended to the model input not affecting the outputs and conclusions at all. The authors attempt to isolate the impact of the probe by comparing performance on a randomly initialized model (where performance collapses) however it seems counterintuitive that this would have no impact and there are not better ways e.g. probing around the token the entity organically appears in the context (at least for the cases where it does)

Q1. Could authors more clearly explain around why the entity triple being appended does not affect the output? For example even with a randomly initialized LLM the mean F1 is around 20 (one would expect zero for complete independence)

We break down W1&Q1 into several sub-questions and address them one by one.

(1) Why does a randomly initialized LLM achieve an average F1 score around 20? Is this reasonable?

An F1 score around 20 represents the result of random guessing. Let us illustrate this with a concrete example: suppose your attribution evaluation dataset consists of samples where each contains a query and 100 candidate sentences, with 10 target sentences containing information relevant to the query. Your task is to identify these 10 target sentences from the 100 candidates.

Now consider a "poor" model that completely ignores the semantic content of candidate sentences and considers all of them relevant to the query—essentially predicting all 100 candidates as target sentences. Let's calculate the F1 score in this scenario:

Precision = $\frac{\text{Number of correctly identified targets}}{\text{Number of predicted targets}} = \frac{10}{100} = 0.1$

Recall = $\frac{\text{Number of correctly identified targets}}{\text{Total number of actual targets}} = \frac{10}{10} = 1.0$

F1 = $2 × \frac{\text{precision × recall}}{\text{(precision + recall)}} = 0.182$

Clearly, an F1 score of 0.182 does not indicate that this "poor" model possesses any attribution capability.

(2) Since probes require specific input formats, are the probing results consistent with the model's actual behavioral patterns?

The probe test results demonstrate high correlation with the model's actual output behavior. Additional analysis based on the LLaMA 3.1 model is provided in Appendix F of our paper, with results summarized in Table 1 below. The methods "Duplicate and append," "Insert prompt randomly," and "Tag and append" can be viewed as three approaches to enhance the model's attention to counterfactual knowledge in the context (detailed descriptions are available in Appendix F).

Table 1: Model Response Patterns Under Different Context Conditions

As shown in Table 1, when the context contains only counterfactual documents, 54.6% of responses follow counterfactual knowledge, significantly higher than responses following factual knowledge (22.7%). This phenomenon aligns with Figure 4(c) in our paper, where the information intensity of counterfactual knowledge in the output layer exceeds that of factual knowledge, and the magnitude of information intensity determines the degree of influence that response knowledge has on the output.

When we enhance the model's attention to counterfactual knowledge in the context through three different methods, the model's output does not shift from counterfactual to factual knowledge. This is consistent with our probing conclusion that "knowledge competition does not occur across categories"—that is, enhancing attention to contextual knowledge does not affect the routing of memory heads to parametric knowledge.

When the input contains both factual and counterfactual documents ($\mathcal{D}\_f\cup \mathcal{D}\_{cf}$ setting), compared to the $\mathcal{D}\_f$ setting, the model's output clearly begins to favor factual knowledge (56.4% vs. 27.7%). Why does the model favor factual knowledge when both types appear simultaneously in the context? Our probing conclusion of "superpositional reconciliation" explains this well. Factual knowledge from the context and factual knowledge from parameters form a superposition, making its information intensity exceed that of counterfactual knowledge sourced only from the context. Since information intensity magnitude directly affects output, the model favors factual knowledge.

Subsequently, when we enhance the model's attention to counterfactual knowledge, the proportion of counterfactual responses increases significantly. The results of this operation under $\mathcal{D}\_f\cup \mathcal{D}\_{cf}$ setting are strikingly different from those under $\mathcal{D}\_f$ setting, but this is not surprising. Under $\mathcal{D}\_f\cup \mathcal{D}\_{cf}$ setting, factual knowledge comes from both model parameters and context. When we enhance the model's attention to counterfactual context, the two compete for the attention of context heads, weakening the context heads' attention to factual knowledge. This aligns with our probing conclusion that "knowledge competition occurs within categories."

(3) Are there better methods? For example, probing around context tokens where entities naturally occur?

We appreciate the reviewer's suggestion to explore more natural probing methods. Indeed, developing better probing techniques to accurately capture models' internal knowledge processing mechanisms remains an important and open research question worthy of continued exploration and improvement. However, the reviewer's suggested approach of "probing at positions where entities naturally occur in context" faces a fundamental limitation, which is precisely the core problem our entity-aware probing framework aims to address: it cannot effectively probe parametric knowledge.

For example, when an external document claims "Trump is the US President," the model's parametric knowledge "Biden" simply does not appear in the input context. We would therefore be unable to probe the model's internal processing and propagation of the parametric knowledge "Biden." Our method enables dynamic tracking of any relevant entity (regardless of whether it appears in the context) by explicitly marking target entities, which is a necessary condition for understanding LLMs' knowledge reconciliation mechanisms.

Thanks to authors for taking time to address/answer some of my concerns. Here are my follow ups

Q1: The example of the .20 F1 baseline seems a bit unconvincing, let me explain why. Authors assume a bad model would predict all positives, thereby coming up with a recall of 1.0. However, using another definition of a bad model could be it predicts only 1 TP in which case precision would be 0.01 and recall would be 0.1; therefore F1 would be 0.01.

Q2: Authors in Q2, do not seem to be answering the essence of my question. As in my question was about whether the probing actually is a window into the model's internal behavior patterns or instead reflects byproduct of the prompting setup and it seems they are talking more about information intensities which they analyzed in the results.

Q3: Authors adequately address why my very quick and dirty idea is not applicable and I agree completely. However, my point was not necessarily to follow that approach (it was meant as a mere simplistic example), but rather to ask whether there are alternative probing strategies that avoid modifying self‑attention through appended triples. So let me better re-phrase: is this strategy the "only" way to probe the model which the authors thought about or has been done earlier in the literature? Did they consider any other approaches, or come across others in previous literature? If so, then why did they choose this approach?

Authors' responses addressed my concerns, no change to my original ratings

Dear Reviewer KnAK,

Thank you very much for taking the time to provide such valuable and detailed feedback on our manuscript. We are pleased that you found merit in the important value of our research on conflict reconciliation mechanisms, our neat and clever counterfactual data synthesis, and helpful probing validation experiments. We address your questions and concerns as follows:

W1. Questions regarding the probe training data and the relationship between entities and their corresponding knowledge types.

The probe training does utilize counterfactual data. We generate the probe's training data based on the prompt template provided in Table 2 of the paper. The prompt includes both "Analysis" and "Counterfactual Data Synthesis" components. The latter generates counterfactual versions of original documents and identifies corresponding counterfactual entities. For the three training data types in the probe, entities map to their knowledge types as follows:

Table 1: Probe Training Dataset Categories

We commit to improving the clarity of section3.3 to avoid misunderstandings.

W2&W6. Concerns about the probe training inputs and whether probes can measure the information strength in residual streams.

Probes are trained on output layer representations, then used to measure information strength in residual streams. This approach is known as early exit [1-3], which has proven to be effective even without special training processes [4], as the residual connections [5] in transformer layers make the hidden representations gradually evolve without abrupt changes.

[1] Fast inference via early exiting from deep neural networks.

[2] Depth-adaptive transformer.

[3] Confident adaptive language modeling.

[4] Bert's output layer recognizes all hidden layers? some intriguing phenomena and a simple way to boost bert.

[5] Deep residual learning for image recognition.

W3. In section 4, eqn 4, which residual stream are you using to compute ?

We compute using the [END] token's output layer representation.

W4. Why use LoRA to implement probes?

As described in line 128 of the paper, we fully follow the approach in [1] for probing entity information, including introducing two new special tokens and a LoRA update. According to the description in Section 4.1 of [1], the "small" LoRA update is designed to help LLMs adapt to changes in input format. We hypothesize that completely freezing LLM parameters and relying solely on two new token embeddings would be insufficient to achieve this adaptation.

[1] Physics of language models: Part 2.1, grade school math and the hidden reasoning process.

W5. Fairness concerns in attribution experiments.

Following the your suggestion, we enhanced the baseline prompting methods with the following conditions to ensure fair comparison:

• +entity: Providing GPT-4o extracted entity information as input.
• +CoT: Incorporating carefully designed CoT-style one-shot demonstrations.

Table 2: Attribution Results with Fair Comparison Conditions vs. Probing Methods

Highlight our motivation: It is important to clarify that our probing method is not intended to compete with prompting methods in terms of performance, but rather to provide a complementary analytical tool for understanding the internal attribution mechanisms of language models.

Validation of Intrinsic Model Capabilities: The results demonstrate that under sufficient conditions (+entity+CoT), instruction-following models can indeed "articulate" accurate attributions. This confirms that models intrinsically possess the ability to perceive entity relevance; they simply require appropriate methods to elicit this latent knowledge.

W7. Concerns regarding alignment between LLM parametric knowledge and HotpotQA contextual knowledge, and suggestions for testing on filtered subsets.

Following the reviewer's suggestion, we filtered the evaluation data to include only questions that LLMs answer correctly without any document prompts. Our main findings remain consistent on these subsets, including distinct routing pathways, within-type competition, and superpositional reconciliation. The primary difference is that factual entities show higher average information intensity compared to the full dataset. Due to rebuttal length constraints, we will include the complete results in the revised paper.

Q1&Q3. Do the context heads and memory heads identified in Figure 3(a) and (b) remain consistent in part (iii) of section 5.2? A bar chart would be a better presentation format.

The context heads and memory heads identified in part (i) from Figure 3(a) and (b) remain consistent in part (iii). There is only one minor exception: the 11th context head in layer 14 falls out of the top 10 in terms of information intensity in subplot (d) "$\mathcal{D}_{cf}\cup \mathcal{D}_f \rightarrow \mathcal{E}_f$". Due to rebuttal policy constraints, we cannot provide graphical results, so we are using tabular data as a substitute. The context heads and memory heads are ranked according to their information intensity from high to low as shown in Figure 3(a) and (b), respectively. We will supplement the results with bar charts in the revised version of the paper.

Table 3: Distribution and Information Intensity Values of Context Heads

Table 4: Distribution and Information Intensity Values of Memory Heads

Q2. Impact of Factual Knowledge in Context on Memory Heads

Based on Table 4's analysis of 10 memory heads, we address this concern systematically. Among these heads, one overlaps with context heads (Layer 12, Head 22, as mentioned in Line 251). When context shifts from $\mathcal{D}_{cf}$ to $\mathcal{D}\_{cf}\cup\mathcal{D}\_{f}$, factual knowledge exists both as contextual and parametric knowledge. The overlapping head (Layer 12, Head 22) shows a 4x+ increase in information intensity (0.018 → 0.079), as it captures both parametric knowledge and is highly sensitive to newly injected factual knowledge in context. In contrast, the remaining 9 pure memory heads show much smaller magnitude changes with inconsistent directions (some increase, some decrease). Therefore, we conclude that factual knowledge in context has minimal impact on pure memory heads, at least on average.

Q4. The range of prauc scores in Fig. 4a and 5a is not the same.

The ranges are indeed different, which is an inevitable result of contextual knowledge competition. In Figure 4a, the counterfactual entity monopolizes all attention from the contextual heads. In Figure 5a, factual documents are introduced into the context, where factual entities draw away some of the attention from contextual heads. Therefore, the range of PRAUC scores becomes smaller. This can also be observed in Table 3 from response to Q1&Q3. After introducing $\mathcal{D}_{f}$, the attention from 10 context heads was diverted to factual entities, reducing the average information strength of counterfactual entities from 0.0854 to 0.0425.

Q5 How are you aggregating the information intensity across heads?

Measure the aggregated information intensity by probing the output representation of the attention layer (described in line 92 of the paper).

Q6&Q8.: Concerns regarding the formation mechanisms and information intensity of contextual vs. parametric knowledge

In Fig. 4(a), the information intensity of contextual knowledge in early attention updates is already 4-6 times higher than parametric knowledge, representing a substantial gap. This indicates that contextual knowledge emerges in early attention routing, while the complete semantics of parametric knowledge do not yet appear. We hypothesize that parametric knowledge formation requires layer-by-layer abstraction of knowledge fragments until complete semantic units are formed near the output layers.

It's worth noting that residual connections in modern neural networks enable representations to gradually evolve without abrupt changes, making it difficult to observe very large information intensity values.

Q7. Line 303 to 305: Could you please elaborate on this?

This indicates that attention competition occurs only within the same type of knowledge (refer to the analysis of Q4), rather than across different categories (refer to the analysis of Q2).

Dear Reviewer jqeN,

We greatly appreciate your thoughtful feedback on our work! We are encouraged by your recognition of the importance and value of our research topic as well as the wealth of experimental insights. We address your questions and concerns as follows:

Q1. What is the conceptual connection between the relevance discrimination task and knowledge reconciliation? Relevance seems like a surface property, whereas reconciliation implies resolution of conflicting information. Why is the former a good proxy for the latter?

The dynamic changes in entity relevance (i.e., information intensity) across layers directly reflect the model's mechanism for reconciling conflicting information, rather than being a superficial property unrelated to the reconciliation process. We illustrate how entity relevance can be used to analyze the model's conflict reconciliation mechanism through a demonstrative example.

Consider the query $q$ = "Who is the current U.S. President?" Suppose the model's parametric knowledge is $\mathcal{E}_p$ = "Biden" while the external document $D$ contains contextual knowledge $\mathcal{E}_c$ = "Trump". The model now faces a decision: which of these conflicting pieces of knowledge, $\mathcal{E}_p$ or $\mathcal{E}_c$, should be used to answer query $q$?

• If only one type of knowledge can be detected in the attention layer's output, i.e., Attention("Biden", "Trump") → "Trump", this indicates that the attention layer reads prompt=$\mathcal{D}+q$ but only routes one type of knowledge for answer generation. In this case, conflict reconciliation occurs at the attention layer.
• If both types of knowledge are routed by the attention layer, but the information intensity of one type rapidly decays in the residual stream at the position where the answer token is generated, i.e., ("Biden", "Trump") → MLP$_i$ → ("Trump"), this suggests that the MLP layer plays the role of conflict reconciler in the residual stream.

(The two hypothetical scenarios above are for illustrative purposes only; the actual conflict reconciliation mechanism is more complex than these two cases. See Section 5.4 of the paper for further discussion.)

W1. Too Narrow Focus on Relevance Discrimination: A large portion of the paper is dedicated to building and evaluating the entity relevance probe. However, relevance discrimination alone does not explain how reconciliation happens. There is a disconnect between the claim of studying reconciliation mechanisms and the design of experiments, which focus more on detection than reconciliation.

In our response to Q1, we clarify how relevance metrics explain reconciliation mechanisms, which we believe addresses your concerns regarding Weakness 1. The experimental design of this paper consistently centers on studying reconciliation mechanisms and provides valuable insights into these mechanisms. This has been acknowledged by Reviewer 6M6N (Strengths 3, 4, 5) and Reviewer oNGe (Strengths 1, 2, 3). Furthermore, probe construction is a necessary means to measure entity relevance (i.e., the information strength of knowledge encoded in model representations), and the probe constructed in this paper differs significantly from classical probes in technical aspects. Therefore, we believe it is necessary to describe the construction method and evaluate its validity. This point has also been recognized by Reviewer KnAk (Strength 3).

W2. Lack of Grounded Real-World Applications: The knowledge conflict scenarios are synthetic and overly simplified (e.g., single-entity substitutions). There’s little discussion of how this mechanism translates to more complex real-world retrieval settings with temporal or causal dependencies.

Thank you for your valuable feedback. We completely agree that the applicability of research work to real-world tasks is an important evaluation criterion. Regarding the concerns you raised, we would like to further clarify the real-world foundations and design considerations of our work:

(1) LLM-based Knowledge Conflict Generation vs. Simple String Replacement:

Our knowledge conflict scenario generation is based on LLM chain-of-thought (CoT) reasoning, which is the widely adopted standard method for constructing knowledge conflict scenarios in current research [1,2,3]. This approach is fundamentally different from the simple rule-based string replacement methods used several years ago [4,5,6,7]. When LLMs construct knowledge conflicts, they consider semantic coherence between contexts, different expressions of the same entity within contexts, and logical associations between multi-hop evidence—capabilities that cannot be achieved through simple string replacement.

(2) Diverse Conflict Types Beyond Simple Entity Substitution:

Our data construction encompasses not only simple entity replacement conflicts but also complex scenarios including temporal and causal conflicts. Specifically, we construct knowledge conflict scenarios based on HotpotQA, which inherently possesses multi-hop reasoning characteristics. Modifications to entities in the reasoning chain or their related attributes directly impact reasoning outcomes. Here is an example of temporal conflict:

Query: Which American rock band was formed first, Soil or Drowning Pool?

Evidence A: Soil, often typeset as SOiL, is an American rock band that was formed in Chicago, Illinois in 1997.

Evidence B: Drowning Pool is an American rock band formed in Dallas, Texas in 1996.

Answer: Drowning Pool.

After analyzing the question and related evidence, the LLM modifies the formation times of both bands as follows:

Revised Evidence A: Soil, often typeset as SOiL, is an American rock band that was formed in Chicago, Illinois in 1995.

Revised Evidence B: Drowning Pool is an American rock band formed in Dallas, Texas in 1998.

Since the temporal sequence has been reversed, this affects the answer judgment:

Revised Answer: Soil

(3) Trade-off Between Synthetic and Real Data:

Relying entirely on collecting real-world conflict samples would be prohibitively expensive. Although synthetic data differs from real data, this represents a feasible and effective solution for batch construction of knowledge conflicts at present.

[1] Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts.

[2] CONFLICTBANK: A Benchmark for Evaluating Knowledge Conflicts in Large Language Models.

[3] Intuitive or Dependent? Investigating LLMs' Behavior Style to Conflicting Prompts.

[4] Context matters: A pragmatic study of PLMs' negation understanding.

[5] BeliefBank: Adding memory to a pre-trained language model for a systematic notion of belief.

[6] Entity-based knowledge conflicts in question answering.

[7] Context-faithful prompting for large language models.

Q2&W3. How does parametric knowledge contribute to relevance judgments? Given that all relevance predictions seem to depend on context (via D), is parametric knowledge playing any functional role in this task?

Since Q2 is a derivative question of W3, we address both concerns together in this response.

The comparison between Figure 4(c) and Figure 5(c) in our paper directly demonstrates the impact of parametric knowledge on relevance judgment. In Figure 4(c), the lines ($\mathcal{D}_{cf}\rightarrow\mathcal{E}\_f$) and ($\mathcal{D}\_{cf}\rightarrow\mathcal{E}\_{cf}$) represent the probing of factual knowledge $\mathcal{E}\_{f}$ and counterfactual knowledge $\mathcal{E}\_{cf}$ from the counterfactual context $\mathcal{D}\_{cf}$, respectively. Although factual knowledge $\mathcal{E}\_{f}$ is never mentioned in the counterfactual context $\mathcal{D}\_{cf}$, we can still observe rapid growth in factual knowledge information intensity in the residual stream near the output layers. The only possible source of this information intensity is the injection of parametric knowledge from the model.

From the perspective of information intensity magnitude, the factual knowledge in Figure 4(c) is weaker than the counterfactual knowledge. This aligns with the LLM's actual response behavior, as shown in Table 1, where 54.6% of model responses follow counterfactual knowledge when the prompt contains only the counterfactual document $\mathcal{D}\_{cf}$. However, when both counterfactual document $\mathcal{D}\_{cf}$ and factual document $\mathcal{D}\_{f}$ are provided in the prompt, the situation reverses. As shown in Figure 5(c), the $\mathcal{E}\_{f}$ from parametric knowledge injection and the contextual knowledge $\mathcal{E}\_{f}$ described in the factual document form a superposition, with information intensity directly exceeding that of $\mathcal{E}\_{cf}$. This also corresponds to the model's actual behavior in Table 1, where 56.5% of model responses follow factual knowledge, far exceeding the 27.7% that follow counterfactual knowledge. Without the information intensity from parametric knowledge injection, the difference between the two would not be so significant.

Table 1: Model response behavior statistics under different document input settings. Data from experiments in Appendix F of the paper.

In fact, the training data for the probes also includes predicting the relevance of factual entities based on counterfactual documents—entities that are not mentioned in the documents. Therefore, the training of relevance prediction does not rely entirely on the alignment between entities in $D$ and the query $q$.