Knowledge Localization: Mission Not Accomplished? Enter Query Localization!

We thank the reviewer for their careful reading and constructive suggestions. We will address each point below.

W1

However, some paragraphs lack smooth transitions.

After carefully reviewing our paper, we have added the following transitions to improve flow. In the revised PDF, these new additions are marked in blue:

1. Lines 223-224, between subsections 2.1 and 2.2 (after proving $K_I$ through statistical evidence): "Beyond statistical analysis, we further validate the existence of $K_I$ through knowledge modification experiments."
2. Lines 298-299, between sections 2 and 3 (after demonstrating KL's limitations): "Next, we explore a more realistic alternative."

W2

Certain experimental results are omitted in the analysis

This weakness appears to align with Q2.

Q2: Table 2 presents metrics such as Reliability, Generalization, and Locality, but not all of these metrics are analyzed in the paper.

We have enhanced Table 2 with additional analysis:

1. When analyzing $\mathcal{N}_i$: "Despite high Reliability and Locality scores on original and unrelated queries, the poor generalization reveals the limitations of this method."
2. When analyzing $\mathcal{N}_u$: We have added that the approach "causes Locality to drop from 0.80 to 0.50", alongside the PPL increase analysis.

These new additions are marked in blue in the revised PDF (Lines 260-261, 265, and 269).

We would like to further clarify that we have analyzed Generalization (Lines 257), and our existing analysis adequately supports our conclusions. Nevertheless, we appreciate the reviewer's feedback and acknowledge that the additional analysis of Locality and Reliability further strengthens our findings.

Q1

In Table 1, the KII ratio for GPT-2 is lower compared to other models. What causes this discrepancy? Is it because GPT-2 has significantly fewer parameters than the other two models?

We are uncertain about why the $K_I$ ratio for GPT-2 is lower. To answer this question would require an in-depth analysis of the factors contributing to $K_I$ and $K_C$. As noted in our future work section (Section 5, second paragraph), investigating the origins of $K_I$ and $K_C$ remains a valuable research direction. Please treat the following speculations as informal discussions rather than rigorous scientific analysis. We speculate about several potential factors:

1. Model architecture and capacity:

   • Smaller models like GPT-2 might be forced to develop more consistent storage patterns due to limited capacity.
   • Larger models may have the flexibility to store the same knowledge in multiple ways, potentially leading to higher $K_I$ ratios.

2. Pre-training process:

   • Early in training, when neural capacity is more available, frequently appearing facts might be stored more consistently ($K_C$). As training progresses and neural resources become scarce, new knowledge might need to compete for resources, potentially leading to more fragmented storage patterns ($K_I$).
   • The initial form in which a fact is encountered (whether simple or complex) might influence its ultimate storage pattern.

However, these are speculative explanations that would require specifically designed pre-training experiments to verify, which is beyond our current scope.

Note: We assume "KII" in your question is a typo for $K_I$.

Q2

See W2.

We are grateful for the reviewer's recognition and insightful comments. We address each point below.

Weaknesses

The QL assumption adds the attention module to knowledge retrieval, which could potentially increase computational load. However, the paper does not analyze how this change impacts model efficiency. Including a comparison of computational costs and scalability between KL-based and QL-based methods—such as inference time or memory usage—would clarify the impact of attention-based neuron selection.

1. Indeed, the QL assumption introduces additional analysis of the attention module, which does increase computational load compared to the KL assumption that ignores it. To address this concern, we have analyzed the computational overhead by measuring inference time and memory usage. Our analysis involves three scenarios when obtaining KN scores:

   • No operation on attention module (equivalent to KL assumption)
   • Suppressing attention module
   • Enhancing attention module

   These operations follow the same setup as illustrated in Figure 10, where we analyze the knowledge synapses (attention module) under different conditions.

   To quantify computational efficiency, we have added a comparative analysis table (marked in blue in the revised PDF, Lines 1065-1070, Table 6, page 20) that presents runtime and memory usage measurements across 100 random samples using an NVIDIA A100 (80GB) GPU.

2. We would like to emphasize that computational efficiency is not our priority. The purpose of our paper is to understand model behavior. Unlike research on LLM applications where computational efficiency is important, our research area prioritizes understanding the true mechanisms of models. Including the attention module analysis inevitably increases computational cost, but this is a necessary trade-off for a more comprehensive understanding of knowledge mechanisms in LLMs. The original KL assumption's omission of the attention module led to a simpler but incomplete analysis.

Questions

It would be better if further details, such as interpretability analyses (e.g., visualization), on how the Dynamic KN Selection process works can be provided.

We appreciate the reviewer's suggestion regarding more detailed analyses. In response, we have incorporated comprehensive distribution visualizations in Appendix C.2 of our revised PDF (marked in blue, Lines 1054-1062, with results shown in Figure 11, page 20). These visualizations specifically analyze the neuron activation patterns in LLaMA3-8b across our complete dataset of 27,610 facts. The configuration aligns with Figure 6's bar chart layout, but instead of showing averaged values, we now present violin plots to illustrate the full distributions. This results in 12 distinct distributions for LLaMA3-8b, covering:

• $K_C$ (Consistent Knowledge): two operations (Sup/Enh) × three neuron types
• $K_I$ (Inconsistent Knowledge): two operations (Sup/Enh) × three neuron types

These distributions provide a comprehensive view of how different neuron types respond to Knowledge Synapse manipulation across our entire dataset, extending the average-based analysis in Figure 6.

We appreciate the reviewer’s thoughtful comments. We will address each point below.

W1

The identification of consistent versus inconsistent knowledge relies on specific thresholding, ... robustness claims.

We appreciate the reviewer's concern about threshold sensitivity. However, we can assert that specific thresholding techniques do not affect the robustness of our conclusions. Here's why:

1. Our primary objective is to confirm the existence of inconsistent knowledge ($K_I$). To establish this rigorously, we implement two key relaxations:

   • Modifying Equation(1) to count any recurring knowledge neuron, which increases KN-Consistency Score ($CS$, in Equation 1) values.
   • Using deliberately low thresholds (static: 0.1, Otsu: 0.08-0.17, In Table 4).

   These relaxations ensure that identified $K_I$ (with low KN-Consistency Score) is indeed inconsistent knowledge. We validate this through multiple knowledge localization methods, model architectures, and T-test statistical validation.

2. Our conclusion remains robust when treating threshold as a variable, as shown in Figure 3 (and Figures 7,8 in appendix). The red threshold line can be moved, showing that while the proportion of $K_I$ may change, its existence remains constant.

   Additionally, following the reviewer's valuable suggestion, we have conducted additional threshold sensitivity analysis (marked in blue in the revised PDF, Appendix C.1, Lines 910-917). The results are presented in Figure 9, page 19. Testing thresholds from 0.04 to 0.80 shows $K_I$ persists even at the lowest threshold.

W2

Although the QL assumption ...large-scale knowledge modifications.

We address this with additional evidence and analysis:

1. Our experiments were conducted on a substantial dataset of 27,610 facts, and the triple-based editing format used in our experiments directly aligns with common knowledge graph modifications in practical applications.

2. To further demonstrate the real-world effectiveness, we have conducted new experiments on sequential editing. (marked in blue in the revised PDF, Appendix C.3, Lines 1072-1092, Table 7)

   Experimental Setting:

   • Random sampling of 100 facts for editing
   • 5 independent runs to ensure stability
   • Model: LLaMA3-8b
   • Methods compared: KL-based ($\mathcal{N}_i$) and our QL-based approach

   Results (mean ± std across 5 runs):

   Method	Rel	Gen	Loc	Avg	ΔPPL
   KL ($\mathcal{N}_i$)	0.32±0.23	0.08±0.12	0.48±0.14	0.29±0.16	0.37±0.24
   QL (Ours)	0.45±0.22	0.41±0.23	0.39±0.10	0.42±0.18	0.29±0.18

   The results show that our QL achieves better overall performance (higher Avg) with less model disruption (lower ΔPPL) than KL.

3. Regarding the reviewer's concern about 'efficiency': For a fact with multiple neighbor queries:

   • KL would require editing each query sequentially compared to QL's one-time editing of the entire fact, thus, QL could actually be more efficient.
   • Although editing only a single query under KL would be faster, it would inevitably lead to worse performance, especially in terms of generalization (as shown by the Gen metric).

Q1

We add detailed implementation information in the appendix (marked in blue in the revised PDF, Appendix C.1, Lines 852-863). The implementation for Figure 1 is straightforward: we calculate KN-Consistency Scores for each fact and visualize them using standard plotting code:

data_frame = pandas.DataFrame(all_data)
violin = seaborn.violinplot(x='Label', y='Value', data=data_frame, cut=cut)

where 'Label' represents relations (e.g., P39, P264) and 'Value' represents KN-Consistency Scores.

Q2

We have not considered this scenario in our paper. Our dataset, as shown in Table 5 in the appendix, provides Example data of the ParaRel dataset. Each fact contains only one triple-form factual knowledge. While multiple knowledge pieces are possible in real scenarios, this makes the problem much more complex and belongs to future work.

However, we have some preliminary thoughts on potential approaches to address the reviewer's questions:

1. "Do they activate their according neurons simultaneously?"

   This depends on the overlap between knowledge neuron sets (let's say KN1 and KN2) of two facts. If KN1 and KN2 have low overlap, they would activate independently and simultaneously.

2. "Is there an additive effect on knowledge neurons?"

   This question becomes relevant when KN1 and KN2 have high overlap. Two scenarios are possible:

   • The activation values remain similar to single-fact cases, suggesting no additive effect
   • The activation values increase, suggesting an additive effect. In this case, we would need to further investigate whether there exists a "saturation" phenomenon in the activation strength.

We believe this case-by-case analysis would be beneficial for understanding these more complex scenarios.

Due to character limits, please see our next response for Q3.

Dear Reviewer,

Thank you for your valuable feedback. We have submitted our rebuttal and sincerely hope we have adequately addressed your questions. We look forward to your response.

Best regards

We appreciate the reviewer’s thoughtful comments. We will address each point below.

W1

The motivation behind challenging the KL assumption could be further elaborated. For example, from an application perspective, it would be beneficial to explain the practical significance of proving the KL assumption's limitations and demonstrating the advantages of the new assumption.

We appreciate this suggestion and will enhance the clarity of our motivation. We have added text in the introduction to better emphasize the practical significance (marked in blue in our revised PDF, Lines 40-42): "Not only them, but also many works have adopted KN theory and applied it to study downstream tasks (Chen et al., 2024b;a; Wang et al., 2024b). Therefore, the study of the theory itself is important."

Furthermore, we would like to highlight that our paper already provides a well-structured motivation. Specifically, in the first two paragraphs of the introduction, we present a clear progression: while KN theory is widely adopted (first paragraph), its fundamental assumption (KL assumption) has important limitations (second paragraph). Figure 1 provides a concrete example demonstrating facts that do not conform to this assumption, illustrating why challenging this assumption is necessary.

Regarding the application perspective, we actually discussed this in Lines 38-40, where KN-inspired model editing methods are a practical application of KN theory. Re-evaluating the fundamental KL assumption naturally impacts these KN-inspired methods. This connection is further demonstrated in subsection 3.3, where we improved existing model editing methods using our proposed QL assumption.

W2

The performance gap between the proposed consistency-aware modification method and the two baseline approaches appears to be minimal, which may reduce the impact of the proposed method.

We respectfully disagree that the gap is small, especially for inconsistent facts ($K_I$). As detailed in subsection 3.3, Findings (1), with supporting data from our Table 3, our method achieves good and more balanced results in both Average and $\Delta$PPL metrics, while previous methods either have low Average or high $\Delta$PPL. To further illustrate with data (also mentioned in our original paper):

For LLaMA3+$K_I$ under the Erasure setting:

• Our method: Average=0.51, $\Delta$PPL=0.09
• Baseline ($N_i$): Average=0.43, $\Delta$PPL=0.05
• Baseline ($N_u$): Average=0.42, $\Delta$PPL=1.05

Since successful editing requires both high Average and low $\Delta$PPL, our method clearly demonstrates superior performance.

The reviewer's misunderstanding may come from the small gap in consistent knowledge ($K_C$). This is actually reasonable (subsection 3.3, Findings (2)), because if all facts belong to $K_C$, the QL assumption degenerates into the KL assumption. That is, KL is a simplification of QL. The purpose of this paper is to prove the existence of $K_I$ and the limitations of the KL assumption, which are essentially equivalent. Since $K_C$ appears when neighbor KNs are consistent, it is naturally a special case of $K_I$, and KL is a simplification of QL. Therefore, for $K_C$, both KL and QL assumption are effective. This actually further demonstrates the value of our QL assumption - it is not a complete overthrow of the original assumption, but rather a superordinate coverage: facts that can be analyzed by KL can be completely analyzed by QL.

W3

The limited number of relations used in the experiments may constrain the generalizability of the findings, making the results less convincing.

We respectfully disagree with this concern. Our experiments are comprehensive and demonstrate good generalizability:

1. Dataset Scale: Although our dataset contains 36 relations, it includes a substantial total of 27,610 entries. We have added this information to the dataset introduction section in the appendix(marked in blue in our revised PDF, Line 841). Moreover, as shown in Figure 3, our findings hold consistently across all relations. Specifically, facts with low KN-Consistency Score (i.e., $K_I$) exist in every relation type. This consistent pattern across our full dataset, which wasn't cherry-picked but based on the original dataset, strongly suggests our findings would generalize to new relations.

2. Robustness Verification: We conducted extensive experiments to ensure the robustness of our findings:

   • Multiple models were tested
   • Different knowledge localization methods were evaluated
   • Statistical significance was verified through T-tests
   • Various thresholds were examined

These comprehensive validations across different dimensions (data, models, methods, and statistical tests) support the generalizability of our conclusions.