Capability Localization: Capabilities Can be Localized rather than Individual Knowledge

Thank you for the positive recommendations and valuable feedback!

Q1: Generate samples utilizing the GPTJ model and validate the localization method on the GPTJ model

A1: We generated 1000 test samples utilizing the GPTJ model and validated the localization method on the GPTJ model. The results obtained are as follows. We have constructed three unified evaluation indicators, including the accuracy of knowledge localization (Overlap), the granularity of identified parameters (Neuron), and the Impact on model Performance when manipulating identified Parameters (IPP). The calculation formula for this IPP is: Performance (parameters for operation positioning) - Optimal performance (random parameters for multiple sampling).

The results showed that our localization method demonstrated similar low localization to previous localization methods, validating our previous conclusion.

Q2: Verify the semantic similarity of 1000 samples

A2: To verify the semantic similarity of the new samples, we adopted the following two methods for validation. (1) The Llama38b model was used to evaluate language similarity, and the score obtained was 94.3%. (2) The semantic similarity formula (Cosine similarity between vectors) was directly used for calculation, and the score obtained was 91.6%. The experimental results indicate that the new samples have high semantic similarity and meet the requirements of subsequent experiments.

Q3: Why hasn't the CNL method been evaluated on a unified dataset

A3: The CNL is a method utilized to locate common neurons, specifically, it captures commonalities in batch data with similar types (such as programming-related datasets). However, the previous method focused on locating individual knowledge, which we have proven to be unreasonable in section 3. Importantly, we have constructed a unified evaluation index in A1, which also proves the effectiveness of the CNL.

Q4: Is there a claim the authors are making with respect to Table 2

A4: Table 2 mainly indicates that the positioning method CNL has high accuracy. Compared with an equal amount of random parameters and 99.85% non-localization parameters, only fine-tuning 0.15% of the parameters can significantly improve the performance of the model, which proves that CNL accurately locates the ability neurons. We have added an additional experiment, as shown in the table below, utilizing the located neurons for fine-tuning (training data is GSM8K) will cause minimal damage to the performance of other tasks (emotion, imdb, and code25k datasets), and may even benefit other downstream tasks. This provides a new paradigm of "Localization+Fine-tuning" for downstream task applications, greatly increasing practical value in real-world scenarios.

Q5: What is the difference between knowledge and capabilities

A5: Individual knowledge includes factors such as language, style, semantics, etc., and it is difficult for individual knowledge to be directly mapped to parameters. However, the capability of the model is to learn a large number of sample summaries and inductions, similar to embedding, which is abstract and has the potential to correspond directly to parameters. In addition, our additional experiments have shown that operational capability neurons significantly affect the performance of the current task, which also proves the correctness of studying the localization of capability neurons.

(1) Enhance experiment.

llama2-7B

GPTj-6B:

(2) Erase experiment.

Llama2-13B

Q6: Fig 2 is missing its x axis label. I am guessing it is the layer number?

A6: Yes, we have updated the figure in rebuttal revision.

Ref

[1] Knowledge neurons in pretrained transformers. EMNLP 2022.

[2] Locating and editing factual associations in gpt. NeurIPS 2022.

[3] Knowledge Circuits in Pretrained Transformers. NeurIPS 2024.

Thank you for the additional experiments and clarifications. I am inclined to keep the score as it is. Please include the updates in the revised manuscript. As a future work, you could consider extending this framework to improve the interpretability of what individual neurons are selective for. For instance: Guertler et al. 2024 https://openreview.net/forum?id=01ep65umEr. All the best!

Q5: What capability does the author refer to.

A5: The capability refers to the ability of model to solve a specific problem. For example, in the GSM8K dataset, the neurons we locate primarily consists of mathematical and multiple-choice question-solving capability, along with some other minor capabilities such as English understanding.

Additionally, Fig. 5 indicates that commonality can be inferred by part of the data. This is also a characteristic of capability, that is, we can determine what capabilities are needed to solve the task of the entire data set based on a subset. In Table 2 and Table 3, we demonstrate that the performance of commonality neurons is related to the model, which means it's connected with the model's capability on the current dataset. Fig 6 shows that enhancement and suppression of commonality neurons can also result in increased and decreased performance on similar datasets. This is also consistent with our conclusion: capability is an attribute that is independent of the specific form of data.

Q6: About the result of Commonality across datasets.

A6: The purpose of conducting cross-dataset experiments is that the model's ability can be cross dataset, and only by achieving significant performance improvements on other datasets, we can clarify that common neurons are the set of corresponding abilities.

We have conducted additional experiments on GPT-J to enhance the persuasiveness of our conclusions. Meanwhile, we provide more detailed explanations of the experiment results in A4. Additional across datasets experiment is in APPENDIX H of the rebuttal version.

Q7: Other questions about the paper

A7: (1) We have increased the clarity of Fig 2 in the article.

(2) The purpose of section 4.2 is to explore potential research directions beyond locating individual knowledge. Interestingly, the experimental results indicate that the localization of capability neurons is valuable, and provides a foundation for the localization of capability neurons.

(3) The $P_s$ in Formula (8) is the set of parameters for localization.

(4) We have revised the terminology incoherent with scientific rigor.

We promise that we have fixed the above issues in the rebuttal version.

Ref

[1] Knowledge neurons in pretrained transformers. EMNLP 2022.

[2] Locating and editing factual associations in gpt. NeurIPS 2022.

[3] Knowledge Circuits in Pretrained Transformers. NeurIPS 2024.

Thank you for your valuable feedback and for recognizing the novelty of the our method. Below, we address some of the weaknesses raised:

Q1: Why it is relevant to localize phenomena at a much coarser level

A1: We have constructed three unified evaluation indicators, including the accuracy of knowledge localization (Overlap), the granularity of identified parameters (Neuron), and the Impact on model Performance when manipulating identified Parameters (IPP). The calculation formula for this IPP is: performance (parameters for operation positioning) - optimal performance (random parameters for multiple sampling).

From the perspective of consistency (metric Overlap), the rewritten individual knowledge is not faithful to the original localization results, and the overlap of the model's capability localization can reach 96.4%. From a granularity perspective (metric Neuron), the number of parameters occupied by a single knowledge is unreasonable (for example, ROME's Neuron reaches 16.7%). From the perspective of operational localization parameters (metric IPP), the impact of operating a single knowledge localization neuron on knowledge is not significant, while the impact of capability neurons on capability can reach 20.0%, which also proves that the localization results of a single knowledge are inaccurate.

Individual knowledge (or a prompt), includes factors such as language, style, semantics, etc., and it is difficult for Individual knowledge to be directly mapped to parameters. However, the capability of the model is to learn a large number of sample summaries and inductions, similar to embedding, which is abstract, so it has the potential to directly correspond to parameters. For example, our experiment below shows that enhancing capability neurons improves other tasks that are not involved in training:

The result have shown that operational capability neurons have a relatively small impact on the performance of the model on other tasks, which also proves the correctness of studying the localization of capability neurons.

Q2: Further discussion on attribution scores and comparison with existing methods

A2: The previous attribution score (KN) referred to locating individual knowledge, while CNL used integral gradients to obtain the common characteristics of batch knowledge and corresponding parameters. To our knowledge, CNL is the first batch knowledge-based localization method. We listed comparative experiments in A1, and the excellent performance proved that previous research (individual knowledge localization) was unreasonable, which also brought new perspectives to subsequent research: focusing on the localization of model capabilities.

Q3: It is not clear from the results if "Existing technology cannot localize individual model parameters" or if "Individual knowledge cannot achieve parameter localization"

A3: We believe that existing methods cannot accurately locate individual knowledge. However, certain inspirations can be drawn from our experimental results: about 0.15% of the model parameters can reflect a group's capabilities. More fine-grained knowledge should be represented by fewer parameters, while the smallest granularity in current methods is neurons. We do not deny that individual knowledge can be located, but we believe that it should be within the parameters of a smaller granularity.

Q4: Inconsistent expression in lines 525-526

A4: We have revised the previous statement and added the following enhancement and erasure experiments, which demonstrate a strong correlation between the neurons we locate and their capabilities.

(1) Enhance experiment. The best results are in bold and highlighted means the suboptimal.

llama2-7B

GPTj-6B:

(2) Erase experiment.

Llama2-13B

Thank you for your constructive feedbacks on the paper! We add detailed explanations for the questions asked in the review.

Q1: What does "commonality" refer to, and what is its relation to knowledge localization?

A1: In section 5.4, we point out that commonality is the set of capabilities. For example, previous knowledge localization utilized individual knowledge called "one plus one equals two", and found the parameters in the model corresponding to that knowledge. However, for the model, this knowledge is not only its computational capability but also its English, style and semantic capability. Therefore, in the article, we refer to the set of model capabilities as commonalities, rather than just simple computational capabilities, which is more rigorous.

Q2: Figure 2 is blurry.

A2: We have updated this figure in the rebuttal version.

Q3: $e_i$ seems to be undefined.

A3: The definition of $e_i$ is in line 682 on APPENDIX D. The $e_i$ refers to calculating the edges in graph G. We add an explanation for $e_i$ in rebuttal version.

Q4: What is the key improvement of CNL compared to KN method?

A4: The key improvement is that CNL has acquired the common feature of batch knowledge, rather than individual knowledge. Specifically, KN refers to the localization of individual knowledge, while CNL utilizes integral gradients to obtain the common characteristics of batch knowledge and corresponding parameters. To our knowledge, CNL is the first batch knowledge-based localization method. We have constructed three unified evaluation indicators, including the accuracy of knowledge localization (Overlap), the granularity of identified parameters (Neuron), and the Impact on model Performance when manipulating identified Parameters (IPP). The calculation formula for this IPP is: performance (parameters for operation positioning) - optimal performance (random parameters for multiple sampling).

The excellent performance proves that previous research (individual knowledge localization) is unreasonable, which also brings new perspectives to subsequent research: focusing on the localization of model capabilities.

Q5: What is the conclusion of Table 2?

A5: Table 2 shows that compared to fine-tuning an equal number of random or non-located parameters, fine-tuning the located parameters can significantly improve performance. The experimental results indicate that: (1) The localization method of CNL is effective, as it identifies the parameters of the common characteristics of batch knowledge within the model. (2) The significant impact of a small number of parameters after localization on the performance, proves that commonality (the set of model capabilities) can achieve parameter localization, which has profound significance. We will add this explanation in the revised version.

To demonstrate the generalization of the results, we conducted the same experiment on the GPTJ model, and the results also confirmed our above conclusion. (For convenience, we only list the results under the last epoch here, and specific experimental results are in our updated Table2):

The best results are in bold and highlighted means the suboptimal.

Q6: What is the function of the common neurons found?

A6: Firstly, by leveraging the superior performance of common neurons, we have broken the traditional mindset of individual knowledge localization and provided the correct research direction: common neuron localization.

Secondly, the cross dataset experiment in Figure 6 demonstrates that we only need to fine-tuning 0.15% parameters to significantly improve or suppress the model's capability in a certain field. This provides a low-cost and efficient solution for applying the model to downstream tasks.

Finally, as shown in the table below, utilizing the located neurons for fine-tuning (training data is GSM8K) will cause minimal damage to the performance of other tasks (emotion, imdb, and code25k datasets), and may even benefit downstream tasks. This provides a new paradigm of "localization+fine-tuning" for downstream task applications, greatly increasing its practical value in real-world scenarios.

Ref

[1] Knowledge neurons in pretrained transformers. EMNLP 2022.

[2] Locating and editing factual associations in gpt. NeurIPS 2022.

[3] Knowledge Circuits in Pretrained Transformers. NeurIPS 2024.

Thank you for your prompt response. Below, I will further elaborate on the question you have raised, highlighting our novel contributions and advantages.

Reply to Q1: The capability of neurons responsible for localization using different corpora is different. For example, in Chapter 5.2, our CNL localization method utilizes the GSM8K dataset (math related) to locate neurons. The capability of the neurons we locate is mainly computational, followed by secondary abilities such as English and style, depending on which capabilities of the models are mainly called upon by GSM8K. Our experiment in Figure 6.a further confirms this conclusion, and after enhancing the localization of neurons, the performance of the model on the Meta_math dataset (math related) is significantly improved compared to other unrelated datasets. This indicates that the neurons we locate are primarily responsible for computing capability.

As mentioned above, our located capability is related to the corpus. Specifically, the located capability is mainly the capability of the model to call upon these corpora when encountering them. As shown in Figure 3, based on the method of controlling variables, we control the capabilities called by the model by setting the main parts and replaceable parts, which achieves the decoupling of different capabilities. However, we have demonstrated in Section 3 that utilizing individual sample for localization (previous methods) is not feasible because the ability of individual sample is not significant. Our CNL has achieved accurate localization of computing capability utilizing batch samples (such as GSM8K). Strictly, the neurons we locate inevitably contain some secondary abilities (such as English, style, etc.) because GSM8K is an English corpus, which is also emphasized in line 527. What we locate is a set of abilities, mainly computing capability ( for the GSM8K dataset).

Reply to Q5: Thank you for your suggestion, we have adopted it. It should be noted that in Section 2, we introduced the current mainstream methods, namely Distributed Parameters (KN), Parameter Layers (ROME) and Parameter Chains (KC). The required corpus for ROME and KC is strict (in the form of triplets (s, r, o)), and they require the position of the last token of the subject, which does not apply to free-text widely present in real-world scenarios (such as the GSM8K dataset in Table 2). We provide the results of KN as a baseline, as shown below

(1) Llama2-7B

(2) GPTJ-6B

The experimental results show that our method is significantly superior to KN. In addition, we have provided unified evaluation metrics in A4 compared to previous methods, which demonstrates the superiority of our approach.

Reply to Q6: Thank you for your keen observation. This is not conflicting. In A6, we mentioned 'may even benefit downstream tasks', the conclusion drawn from evaluating multiple downstream tasks. For example, the value '9.43' in A6 is the average performance growth of other datasets after training separately on GSM8K, Emotion and Code25K. It cannot be denied that there may be performance degradation on a small number of tasks, but the performance damage is minimal. Meanwhile, as shown in the results of A6, our method can minimize the damage to other tasks compared to w/o located (-5.58% and -3.31%), and the experimental effect is significant. We have added specific comparison results in Appendix I, such as Fig 11.e and 11.f. The previous method has caused considerable damage to other downstream tasks (-37%).

Thank you for your response. First, section 4 serves as a pivotal link in our paper. It demonstrates the potential of capability neurons for parameter localization, providing early analysis and inspiration for the capability neuron localization discussed in section 5.

In section 3, we show that previous attempts at individual knowledge localization are unreliable. To further analyze the factors that can achieve parameter localization, we construct a decoupled dataset in section 4, which includes a main part and a replaceable part. By using this dataset to locate neurons, we can better control variables and clarify which factors contribute to parameter localization with significant overlap.

As mentioned in the experimental results on line 321, for the same replaceable part—for example, "1 + 1 = ?"—the average overlap of neurons located for 1,000 pairs of subsample 1 (sub1) and subsample 2 (sub2) is only 15.6%. This indicates that individual knowledge localization is unreliable, even when sub1 and sub2 both contain the mathematical problem "1 + 1 = ?".

Interestingly, when we take the intersection of the localization results from 1,000 sub1 samples, the overlap reaches 7.3%. Unlike the previously mentioned 15.6% (which pertains to two samples, sub1 and sub2), this 7.3% represents the overlap among 1,000 samples. This suggests that the model has considerable potential in localizing computational ability (across all sub1) or programming ability (across all sub2).

Furthermore, section 4 reinforces the unreliability of individual knowledge localization. Even when two individual pieces of knowledge share the same replaceable part, we still do not achieve a high overlap. However, 1,000 different samples that reflect the model's computational or programming capabilities involve 7.3% or 8.6% of the same neurons, respectively. This result is both exciting and enlightening. Therefore, in section 5, we further extend this experiment on datasets reflecting mathematical, programming, and linguistic abilities, demonstrating the feasibility of capability neuron localization.

Finally, regarding the experimental results, the 7.3% overlap is less than 15.6% because 7.3% is the overlap of neurons localized across 1,000 samples—a challenging task. In contrast, 15.6% is the average overlap of neurons localized between sub1 and sub2 with the same replaceable part.

To further demonstrate the effectiveness of decoupling experiments, we constructed decoupling data based on the emotion recognition dataset SemEval [1]. For example, the main part is "William did not complete his homework," and the replaceable parts are "Is this sad or joyful?" and "What should he do next."

We commit to updating this part of the results and explanations in the revised version.

Ref

[1] SemEval-2016 Task 5: Aspect Based Sentiment Analysis. Hal.science 2016.

Q7: The Scalability of CNL Method

A7: Firstly, in terms of computational complexity, as the parameter size increases, the computational resources required for CNL inevitably increase. Our experiment was conducted on models such as GPTJ, LLAMA2 7B, and LLAMA2 13b, requiring only 4 * NVIDIA A100 (40 GB) of computing resources to complete. Compared to previous methods of fine-tuning models with full parameters, the computational resources and time consumed by CNL are negligible.

Secondly, regarding the adjustment of hyperparameters, as shown in Eq 9 and 10, our hyperparameters mainly include threshold σ and S. The threshold σ is based on the standard deviation theory and does not require manual adjustment. Experimental results show that as the parameter size increases, the number of neurons located also increases, and the proportion is similar.

(1) Llama2-7B

(2) Llama2-13B

Finally, regarding the relationship between data volume and parameter size, as the parameter size changes, we also listed the required data volume for different models.

(1) GPTj-6B

GSM8K:

Emotion:

(2) Llama2-13B

GSM8K:

Emotion:

The experimental results indicate that there is no significant difference in the required amount of data at different parameter scales.

Q8: How capability localization could enhance model interpretability in practical applications.

A8: Firstly, we conducted a new experiment to demonstrate that CNL improves the interpretability and applicability of the model in practical scenarios. The three indicators refer to fine-tuning random parameters, w/o located parameters, and located parameters, and evaluating the model after fine-tuning located parameters on other tasks. The results indicate that fine-tuning the located parameters can not only improve the performance of the model in current tasks but also contribute to other tasks, which proves that CNL can improve the performance and robustness of the model.

At the same time, updating only the located parameters is not only more cost-effective, but also demonstrates the accuracy of capability positioning without harming other tasks. By establishing the correspondence between internal parameters and the different capabilities of the model, the transparency of the model has been improved. We promise to provide parameter correspondences for different models, which will help with interpretability research of the community in practical applications.

To our knowledge, CNL is the first method to implement capability localization and has demonstrated that capability can achieve parameter localization rather than individual knowledge, which will contribute to subsequent interpretability research.

Q9: Further explanation on the chart

A9: Figure 1 mainly visualizes the reliability and fidelity experiments, and conducts experimental analysis on the previous three forms of knowledge storage. The red box in Figure 3 indicates that there are differences in the parameters used for localization between subsample1 and subsample2. Figure 4 visualizes the localization parameters within the model, with darker-colored areas referring to the localized parameters. We have added this explanation in the rebuttal version.

Ref

[1] Linear and Nonlinear Models.Fixed effects, random effects, and mixed models. GALE 2006.

[2] Knowledge neurons in pretrained transformers. EMNLP 2022.

[3] SemEval-2016 Task 5: Aspect Based Sentiment Analysis. Hal.science 2016

Thank you for recognizing the importance, effort in method, and applications of our work. We outline our response to the main concerns:

Q1: Unified evaluation indicators and results

A1: Thanks again for your advice! We have constructed three unified evaluation indicators, including the accuracy of knowledge localization (Overlap), the granularity of identified parameters (Neuron), and the Impact on model Performance when manipulating identified Parameters (IPP). The calculation formula for this IPP is: performance (parameters for operation positioning) - optimal performance (random parameters for multiple sampling).

The excellent performance proves that previous research (individual knowledge localization) is unreasonable, which also brings new perspectives to subsequent research: focusing on the localization of model capabilities.

Q2: Choice of σ and S

A2: In statistics, σ=3 is commonly used to detect outliers [1]. The larger the σ, the more stringent the requirements for locating neurons. In order to better validate the effectiveness of our method, we have chosen a more stringent value of σ=6: we generally consider data with σ>6 as outliers in statistics. However, in response to your suggestion, we have also supplemented experimental results under σ = 3. In addition, S=19 is a setting we adopt from the previous method KN [2]. Here we have listed experiments on fine-tuning and localization accuracy with σ=3, and the more stringent condition of σ=6 can achieve considerable experimental results.

(1) Fine-tuning & $\sigma = 3$

(2) Localization & $\sigma = 3$

We have provided experimental results with σ=12 at global rebuttal.

Q3: Extending decoupling experiments to tasks in other fields

A3: To further demonstrate the effectiveness of decoupling experiments, we constructed decoupling data based on the emotion recognition dataset SemEval [3]. For example, the main part is "William did not complete his homework," and the replaceable parts are "Is this sad or joyful?" and "What should he do next." The experimental results are as follows

The results indicate that the conclusions in the paper are still met on the emotion recognition dataset, and we will add these results in the rebuttal version.

Q4: Statistical evaluation of enhancing local neurons to improve performance

A4: We extended the experiment in Table 2 to the GPTJ model, and the results showed that enhancing local neurons on different model architectures can still improve performance. The term 'most sensitive' here refers to the performance improvement that can be achieved by fine-tuning only 0.15% of the located parameters compared to random parameters.

The best results are in bold and highlighted means the suboptimal.

Q5: About the random-neuron conditions in the “enhance” and “erase” experiments

A5: Your suggestion was very helpful, and we have adopted it. We averaged the results through three random experiments and have updated the experimental results in Table 2 in rebuttal revision. The results once again confirmed the previous conclusion. For “erase” experiments, we also supplemented the results of Llama2-13B:

Erasing localized neurons can lead to more significant performance degradation.

Q6: Regarding providing more control conditions to demonstrate the impact of common neurons

A6: We have updated more comprehensive experimental results in Tables 2 and 3, sampling and averaging three groups of random neurons in rebuttal revision. The experimental results demonstrate the significant impact of localization parameters on the model's capability.

The reviewers have addressed a number of my concerns. I am happy to increase by score to 6. I do request the authors to spend more time describing the figures and adding figure legends. The paper is quite dense and hard to follow in its current form.