What does the Knowledge Neuron Thesis Have to do with Knowledge?

Thank you so much for the insightful review!

We agree that the results presented in the first paragraph of section 3.2 are overly compact. We have reflected that and we will break it into several paragraphs in the updated version of the paper.

Table 1 presents our own results from our newly proposed criteria and datasets. Figure 6c was Yao et al.'s (2023) evaluation. We will include a citation in the figure caption as well to avoid the confusion.

• Q1: Future interpretability research directions.

  In our opinion, we think the recent work that tries to identify “circuits” in PLMs is a promising direction. Our paper shows that the recall of facts and the expression of linguistic phenomena may follow similar mechanisms but they are certainly much more complex than a simple key-value dictionary.

  However, the community is still at a very early development stage in creating this circuit mode of interpretation. The circuit identification methods are ad hoc and can only be applied to a small set of tasks. In future work, we will try to formalize the circuit interpretation framework and apply it to more tasks and phenomena.

  We will extend our discussion to this in the updated draft. And we would like to thank the reviewer for reminding us of the circuit interpretation references.

• Q2: Missing references.

  Thank you for also reminding us of the Clark reference. We will include it, as well as the circuit references in the updated version of the paper.

Thank you for your review and the recognition!

Thank you for the feedback on the flow of the paper. We will reword the end of section 3 and section 4 to improve continuity.

Thank you for the great review and the insightful feedback!

However, we want to respectfully disagree with the reviewer's assertion that our work is studying LLMs on a "cognitive" level and not suitable for ICLR. Our paper is about interpreting and understanding the function of each transformer component to bring direct model editing of LMs to fruition. If the knowledge neuron thesis were to hold, direct model editing could be implemented by simply editing the MLP neuron activations or weights. This will significantly enhance the safety and privacy aspects of LLMs. But as we proved in this paper, such a simple fix does not exist. The limitations of current KN-based model editing methods originated from the limitation of the framework that only edits MLP neurons. Future improvement with only simple neuron manipulation may be a futile endeavour. We must resort to a more intricate means of interpretation and manipulation beyond the individual neuron level. Therefore, our work is about interpreting the representations within the transformer architecture learned by LLMs, which has more of a focus on the architecture of transformers than linguistics. This is also why we chose to submit to ICLR rather than an *ACL conference, despite similar deadlines.

The misconception may have arisen from our citation of formal vs. functional competence work from cognitive science. In fact, in this respect, we are "anti-cognitive," because we dispute such appeals to pseudo-cognition in mechanistic interpretability work. In this paper, we disputed two such points in prior work, i.e., Dai et al.'s (2022) attribution of layer depth to the distinction between syntax and semantics, and both Dai et al.'s (2022) and Meng et al's (2022, 2023) strident use of the term "knowledge." In the paper we called for grounding the interpretation of the computational mechanisms of LLMs to robustly defined, human-interpretable tasks such as syntactic phenomena rather than broad analogies to human cognition.

Lastly, we wish to point out that the ICLR references we have cited in our paper are very much in the same vein as our work, and so there shouldn't be a question as to the suitability of our paper to ICLR. Our work re-examines current mechanistic interpretability work (Meng et al., 2023) and, as we have discussed with reviewer @7T91, we concur with the future trend of interpreting LMs with "circuits" (Wang et al., 2022). So both sets of papers — the past and future of our research programme — were published at ICLR.

• Q1: Neuron selection procedure.

  The KN thesis hypothesizes that "knowledge" can be localised to a small number of neurons. There are two parts to this thesis: 1) the representation is localised to a small number of neurons, and 2) the representations that LMs capture encode "knowledge." Our paper confirms and extends the first part of the thesis to syntactic tasks, but refutes the second part — the representation localises complex patterns, but cannot, by any commonly accepted definition, constitute knowledge. We did not change the neuron number limit because the MLP weights can indeed still be interpreted as patterns in a localised fashion.

  We also strongly agree with the reviewer's intuition that knowledge can reside in the network in a much more distributed fashion, and this is the message we are trying to communicate. We further emphasize this in the conclusion of the paper. It is obstinate and unreasonable to claim that LMs do not capture any knowledge at all, yet it is oversimplistic to adopt the view of the KN thesis to claim that knowledge is directly stored in the MLP weights just as in key-value dictionaries. Future breakthrough in mechanistic interpretability must adopt a more distributed approach that looks beyond the MLP weights. We are dedicated to moving towards that goal in our future work.

• Q2: Consistent messaging.

  As the reviewer has summarized, the main point of this paper is that "the existence of syntax-knowledge neurons in LLMs analogous to fact-knowledge neurons, and that neither sets of knowledge neurons can be considered to robustly represent 'knowledge'." This is a wording issue. We re-examined the paper and have updated the wordings to make it more consistent. For example, the sentence on page 7 is reworded as "LMs solve the two types of tasks (syntactic and factual) using the same mechanism."

• Q3: A "contributions" section.

  Such a good point! We will add such a section.

• Q4: Illustration of other linguistic phenomena.

  Because of space limitations, we must place the other linguistic phenomena (subject-verb, gender and number anaphor agreement) in the appendix.

• Q5: Definition of formal and functional competence.

  Thank you for the suggestion. This will certainly provide the readers with more context. We will move the brief definition of formal and functional competence to the introduction where it first appears.