Can Neuron Activation be Predicted? A New Lens for Analyzing Transformer-based LLM

1. The experiments are conducted only on LLaMA-2 and GPT-J, which may limit the generalizability of the findings. It would be beneficial to see experiments on a wider range of transformer-based models to validate the effectiveness of NPL.
2. The method lacks novelty, which essentially is a linear regression between neuron activations of different layers.

Major:

• Experiment results: The PPL of original LLAMA-2 and GPT-J (33.08 and 26.58 in Table 2), on wikitext2 are too high regarding their large number of parameters. For example, GPT-J (6B) is known to be better than GPT-2 (1.5B) (check https://huggingface.co/EleutherAI/gpt-j-6b) , but GPT-2 (1.5B) achieved 18.34 PPL on wikitext2. I guess the trained large language models did not converge well, or maybe the simplified forward propagation (no consideration of layer normalization) is the key reason.
• Lack of explanations: I believe that this paper could be improved by explaining more clearly some parts as follows: : In section 3, to select which dimension of state to substitute, this paper used L2 distance and Pearson correlation. However, it is unclear what are the two objects used to compute distance (or correlation). : Equations 7, 8 are introduced without explanations of some notations, such as ‘g’ with square brackets, sigma square (maybe variance?), operation ‘|’ with ‘ㄱ’ shape symbol, and ‘theta_P’. : The difficulty score, ‘s^{l,i}’ in Equation 8, is indexed with only i-th neuron and l-th layer. However, because this means difficulty for predictability, I guess it needs additional index to indicate ‘from which layer’s neuron is used to predict’. : For the baseline of Figure 5’s experiments, this paper utilized the difficulty score of a random subset of tokens. Firstly, it is hard to understand what is the random subset of tokens. Also, the difficulty score requires a property function (P) to filter out neurons. This is not explained in the paragraph within lines 384~391. Also, it is not clear why this could be the global baseline for n-gram sensitive, difficulty (loss) sensitive, position sensitive analyses? : Each property functions, such as n-gram and position sensitivity, need clearer definitions. Also, it was hard to understand what ‘arbitrary’ and ‘successive’ positions with only text.

Minor:

• Less intuitive naming for two main perspectives: For example, ‘global view’ (relationship between neuron activation and model performance) and ‘local view’ (relationship between neuron activation and linguistic functions) are less intuitive.
• Location of the paragraph (line 213~215) looks improper in section 3.2, because it contains the contents of Section 4 (Table 1, ‘Complete’, ‘Partial’ substitutions and ‘Original‘ models). It is hard to follow this paragraph.
• Need clearer explanations: For example, about the findings in section 4.2, the authors might want to clearly mention which type of substitution (complete or partial) they are talking about.

• While the NPL framework provides a novel lens, its practical applications, especially beyond interpretability, are not clearly outlined. Integrating this framework into tasks that directly improve model performance or efficiency would enhance the relevance of the work.
• The experiments could benefit from a wider range of models or additional comparisons with other interpretability frameworks. This would clarify the uniqueness and generalizability of the NPL approach across diverse architectures.
• Although the analysis of "background neurons" is insightful, the study of functional neurons is a well-trodden path in neuron interpretability. Incorporating additional layers of functional analysis could further validate the utility of NPL.