Thanks for your comments, we have prepared the following responses to address your concerns.

W-1: Section 3.1.3 is seemingly based on a single prompt.

We design the experiments in Section 3.1 as a qualitative analysis. Although we discuss our observation via one single instance in the main paper, we actually check that it holds for more cases, and we provide their saliency maps in Appendix A.2. We quantify this observation in Section 3.2 by comparing the density of importance scores on instruction words over the followed and un-followed instances. We observe a higher density score on the followed instances. Thus, both visualization (Section 3.1) and quantitative experiments (Section 3.2) demonstrate that instruction words constantly guide the generation process as the key behavior of LLMs to align with human preference. In our revised version, we will emphasize that results in Section 3.1.3 pose a hypothesis and will be confirmed in Section 3.2.

W-2: Expecting results of importance density from LLaMA.

This experiment aims to verify our hypothesis drawn from Section 3.1.3 quantitatively. Thus, the revised version would rephrase this conclusion with a more proper expression that "Table 1 underscores the models generate responses more aligned with human preference by better identifying and harnessing instruction words from user prompts." In addition, we believe it is a good suggestion to compare LLaMA and Vicuna under this setting. Thus, we are conducting the experiment with LLaMA. We will post the results and the rephrased conclusion in our revised version, which will be uploaded soon later.

W-3.1: Expecting more interpretation cases in Table 2.

Table 2 only presents the interpretation of 5 principal components because we cannot include those cases that are hard to present in the main paper within such limited spaces. However, in Appendix C.2 and Figure 7, we demonstrate that over 80% of principal components from the last layer of Vicuna/LLaMA are interpretable by ChatGPT. In addition, Table 9 and Table 10 include more interpretable principal components from both the last and the first layers. Overall, the word lists always show clear and concise concepts to humans. We will release the full annotated dataset once the paper is accepted.

W-3.2: Table 4 only compares the models over 4 concepts.

The categories in Table 4 are not interpreted concepts. They are 4 scenarios and 4 linguistic disciplines used to study the shift of the interpreted concepts after instruction-tuning.

W-4: The main text should mention that ChatGPT is used to identify concepts.

Thanks for pointing this out! The revised version will emphasize in the main text that ChatGPT is considered as the machine annotator to scale up our interpretation method.

W-5: It is unclear how the results show the importance of prompt diversity.

Finding 1 provides evidence that a model is more helpful for humans if it can better identify those instruction words. In addition, Finding 3 demonstrates that self-attention heads would directly learn the word-word relations with instruction words (verbs). Since diverse prompts would offer diverse instruction words, we could expect the model to achieve a better performance by training on diverse prompts.

W-6/W-7: The abstract and introduction could be improved.

Thanks for all these suggestions, such as explaining terms earlier and introducing the research methods more clearly. Our revised version will take them into consideration.

W-8: In the introduction, you say that instruction-tuning updates model weights differently from pre-training, and it's unclear what you mean since pre-training of course, also updates weights.

The original writing wants to point out that although related works have studied the weights of pre-trained models, their findings could not extend to instruction-tuned models since instruction tuning has the chance to modify the pre-trained weights.

W-9: The example sentence in the introduction is not clear enough to the readers.

We want to provide a prompt such as "fix grammar: me and him goes to an stores for buy an apples" to demonstrate the instruction and context parts. In this case, "fix grammar:" is the instruction part, while the rest is the context part. The revised version will describe this case more accurately with more sentences.

W-10: Typos and sentences difficult to be followed.

Thanks for pointing these sentences out! Our revised version would try to improve these sentences.

Q-1: Why do you include Section 2.1? That is very common notation and the Transformer architecture can also be assumed known I'd say.

We include Section 2.1 to provide formal guidance of notations in this work so that the readers could easily follow the methods proposed in different sections. In our revised version, we will further refine this section, and introduce some observations that motivate this study from prior works, such as the "lost-in-the-middle" phenomenon.

Q-2: Why do you use a different set of hyper-parameters for 3.2.2 than for 3.2.1 when you're trying to verify the assumption in the latter with the former?

In our study, L=10 is set as a constant for all experiments. $b$ is a hyper-parameter to consider how weak the importance score could be considered as noise. We intentively set $b$ to 0 because our aim was to present all available information, even if it included some noise, to our readers through heatmap visualizations. We were concerned that the readers might raise objections if we did not provide a comprehensive overview of the salient map, hence the choice of $b=0$.

Q-3: Explain why it is sensible to interpret feedforward networks with PCA.

Section 4 aims to interpret the knowledge encoded in feedforward networks; the most straightforward way is interpreting their neurons (vectors of the weight matrixes). However, many studies have pointed out that neurons are polysemantic [1,2,3,4], leading to a poor interpretability to humans. Therefore, we propose to decompose these neurons (patterns) via principal component analysis and understand the basis of these neurons.

[1] Trenton Bricken, et. al. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Anthropic Blog, 2023.

[2] Arora, Sanjeev, et al. "Linear algebraic structure of word senses, with applications to polysemy." Transactions of the Association for Computational Linguistics 6 (2018): 483-495.

[3] Elhage, Nelson, et al. "Toy models of superposition." arXiv preprint arXiv:2209.10652 (2022).

[4] Cunningham, Hoagy, et al. "Sparse autoencoders find highly interpretable features in language models." arXiv preprint arXiv:2309.08600 (2023).

Q-4: Could you justify the linear approximation of the importance and especially why it's OK to just incorporate the embedding of the input token?

We want to prove that why equation $I_{n,m}=p(y_m|Z_m) - p(y_m|Z_{m,/n})$ could be approximated with the first first-order Tylor extension to $I_{n,m}\approx \frac{\partial f(y_m|z_m)}{\partial \mathbf{E}_i[x_n]} \cdot \mathbf{E}_i[x_n]^\top$.

We rewrite $p(y_m|Z_m)$ as $f(y_m|\mathbf{Z}_m)$, where $f$ is the language model, $\mathbf{Z}_m\in\mathbb{R}^{(N+m-1)\times d}$ are the word embeddings of the input token sequence $Z_m$, and the $d$-dimensional word embeddings of a token $w\in Z\_m$ is defined as $\mathbf{E}\_i[w]$.

Thus, we have $I\_{n,m}=f(y\_m|\mathbf{Z}\_m) - f(y\_m|\mathbf{Z}\_{m,/n})$, where we let the $n$-th row vector of $\mathbf{Z}_{m,/n}$ be zeros.

The first order Taylor expansion of $f(y\_m|\mathbf{Z}\_m)$ around $\mathbf{Z}\_{m,/n}$ is $$ f(y_m|\mathbf{Z}_m)\approx f(y_m|\mathbf{Z}_{m,/n})+\frac{\partial f(y_m|\mathbf{Z}_m)}{\partial\mathbf{Z}_m}\Bigg|_{\mathbf{Z}_{m,/n}}\cdot (\mathbf{Z}_m-\mathbf{Z}_{m,/n})^\top. $$

Since $\mathbf{Z}_{m,/n}$ differs from $\mathbf{Z}_m$ only in the $n$-th row, the term $\mathbf{Z}\_m-\mathbf{Z}\_{m,/n}$ is just the vector $\mathbf{E}_i[x_n]$ and the other rows are zero.

Therefore, the above equation could be simplified as: $$ f(y_m|\mathbf{Z}_m)\approx f(y_m|\mathbf{Z}_{m,/n})+\frac{\partial f(y_m|\mathbf{Z}_m)}{\partial\mathbf{E}_i[x_n]}\cdot \mathbf{E}_i[x_n]^\top. $$ Bring this approximation back to the definition of $I_{n,m}$, we have $I_{n,m}\approx \frac{\partial f(y_m|\mathbf{Z}_m)}{\partial\mathbf{E}_i[x_n]}\cdot \mathbf{E}_i[x_n]^\top$.

The above derivation justifies the approximation of the importance with gradients over the word embedding of the input token $x_n$.

Thanks for giving answer to my questions, for Q1-3 the answers are clear. With this message I want to respond to your answers to some of my weaknesses.

Even if you clarify that Section Section 3.1.3 is just a hypothesis, the plot you use is still very hard to interpret.

For my second point about the phrase "Table 1 underscores that instruction fine-tuned models excel in identifying and harnessing instruction words from user prompts for response generation" you say you will add results, but for now this is still an unsupported claim in the paper.

The fact that you cannot discuss all principal components of table 2 in the main text is understandable, but the reader is now left questioning this without fully going through the appendix. I think even with limited space you should have a summary of the findings presented in the appendix for these kinds of things.

Additionally, I still don't understand after your explanation how the results show the importance of prompt diversity. It seems like you're saying that instruction words help a model in performing tasks, and therefore prompt diversity is important. I don't see how this is related to diversity. You say "Since diverse prompts would offer diverse instruction words, we could expect the model to achieve a better performance by training on diverse prompts.", but I don't believe you can claim this without actually doing that experiment.

Unfortunately, because of the above, I cannot change my current rating. I believe this manuscript needs too much rewriting to be published.

Thanks for your comments, we have prepared the following responses to address your concerns.

W-1: Section 3.1.3 is not convincing since it is made by analyzing the visualization of an instance without quantitative analysis.

We design the experiments in Section 3.1 as a qualitative analysis. Although we discuss our observation with one single instance in the main paper, we confirm it holds for more cases and provide their saliency maps in Appendix A.2. We then quantify this observation in Section 3.2. Thus, both visualization (Section 3.1) and quantitative experiments (Section 3.2) demonstrate that instruction words constantly guide the generation process as the key behavior of LLMs to generate responses aligned with humans. The revised version will emphasize the connection between results in Section 3.1 and Section 3.2.

W-2: Settings of hyper-parameters.

In our study, L=10 is set as a constant for all experiments. $b$ is a hyper-parameter to consider how weak the importance score could be considered as noise. We intentively set $b$ to 0 because our aim was to present all available information, even if it included some noise, to our readers through heatmap visualizations. We were concerned that the readers might raise objections if we did not provide a comprehensive overview of the salient map, hence the choice of $b=0$. For the quantitative analysis in Section 3.2, we used a grid search on $b$ and $p$ to minimize the p-values.

Q-1: I don't understand the derivation of S_{n,m} from I_{n,m} and its intuition.

We derive $I_{n,m}$ to $S_{n,m}$ with two intuitions. Firstly, we consider that $I\_{n,m\_1}$ and $I\_{n,m\_2}$ are not directly comparable since their gradients are computed starting from different confidence scores, i.e. $f(y\_{m\_1}|Z\_{m\_1})$ and $f(y\_{m\_2}|Z\_{m\_2})$. Predicting token m_1 and token m_2 are two word classification tasks with different target label sets. For example, you want to generate the word sequence "this is a short sentence", and let n=this, m_1 = is and m_2 = short. Since "this is" is a very common phrase, while the word "short" is less common in following the word "a", we have $f(y\_{m\_1}|Z\_{m\_1})<f(y\_{m\_2}|Z\_{m\_2})$. To make them comparable across each prediction word, we normalize them from 1 to L as $\left\lceil L \times \frac{I_{n,m}}{\max\_{n^\prime=1}^N I_{n^\prime,m}}\right\rceil$. Secondly, we consider that estimating the gradients over such a large language model may introduce noise. Thus, we sparsify the importance by setting that importance lower than b as 0.

Q-2: Adding the results of importance density $S_{n,m}$ for LLaMA.

Yes, we are conducting the experiment and manually labeling follow/unfollowed for the outputs from LLaMA. The revised version will bring the result with LLaMA.

Misc/typos.

Thanks a lot for pointing out these typos. We will fix them in our revised version. We will also conduct another self-check for those typos.

Thanks for your comments, we address the concerns you raised as follows.

Major-1: It isn't clear what it ultimately delivers to the reader.

This paper contributes novel insights into how instruction-tuned models generate responses aligned with human preferences. This is not fully explored in prior research. While the importance of instruction words seems intuitive, our study empirically reveals how self-attention and feed-forward networks distinctively contribute to this capability. Specifically, the self-attention heads from the bottom layers are responsible for recognizing the instructions by encoding more word-word relations with instruction verbs, while feed-forward networks rotate their pre-trained knowledge toward user preferences.

Our research provides specific recommendations for building models with instruction-following ability. For example, our results suggest that training self-attention heads is more crucial, particularly those in the lower layers, when full-parameter fine-tuning is not feasible due to limited resources. Additionally, for tasks that need a substantial amount of pre-trained knowledge, such as open-domain QA, it's advantageous to train feed-forward networks concurrently. These guidelines are grounded in the findings of empirical studies [1, 2, 3].

Furthermore, our study observed some unexpected phenomena. Contrary to conventional understanding, language models encode a big proportion of semantic knowledge in the bottom layers, while the top layers encode considerably less. Additionally, our findings suggest that models may rely on inherent language modeling abilities, rather than instruction-following abilities, to complete user tasks. We also identified the 'lost-in-the-middle' issue from the pre-trained models, where models focus on the beginning and end of input texts, overlooking the middle sections. These findings are crucial for refining model architectures and training processes.

Moreover, our study introduces the concept of 'importance density' as a new metric for evaluating how models utilize input prompts, with potential applications in model training (e.g., the reward signals for RLHF), backdoor detections, hallucination detections, and so on.

[1] Hu, Edward, et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR. 2022.

[2] Juletx. Alpaca-LoRA. https://github.com/tloen/alpaca-lora. 2023.

[3] Sun, Xianghui, et al. A Comparative Study between Full-Parameter and LoRA-based Fine-Tuning on Chinese Instruction Data for Instruction Following Large Language Model. arXiv. 2023.

Major-2.1: Section 3.1 makes a claim by analyzing the visualization of a single instance.

We design the experiments in Section 3.1 as a qualitative analysis. Although we discuss our observation with one single instance in the main paper, we confirm it holds for more cases and provide their saliency maps in Appendix A.2. We quantify this observation in Section 3.2. Thus, both visualization (Section 3.1) and quantitative experiments (Section 3.2) demonstrate that instruction words constantly guide the generation process as the key behavior of LLMs to generate responses aligned with humans. The revised version will emphasize the connection between results in Section 3.1 and Section 3.2.

Major-2.2: Improving the result presentations of tables.

Thanks for the suggestion, we will take all of them to our revised version. Here are direct responses to your concerns:

(1) Yes, the null hypothesis is the difference between the two values is zero. We want to show that the difference between the experiment and control groups is statistically significant.

(2) The instruction part of an input prompt includes sentences that describe background information (optional) and the required actions for a task. For example, giving the user input prompt "fix grammar: me and him goes to an stores for buy a apples.", the instruction part would be "fix grammar:". The authors manually annotate the instruction parts of 632 samples from the three datasets. Please see Appendix B.1 for more details.

(3) Table 2 only presents 5 interpretation cases out of 69 because we cannot include the cases that are hard to present in the main paper with such limited spaces. In Appendix C.2 and Figure 7, we demonstrate that over 80% of principal components from the last layer of Vicuna/LLaMA could be interpreted by ChatGPT. In addition, Table 9 and Table 10 include more interpretable principal components. Overall, the word lists always show clear and concise concepts to humans and we will release the full annotated dataset once the paper is accepted.

Minor-5: Constructing new vocabulary with the ShareGPT dataset.

The tokenizing pipeline for user prompts from the ShareGPT dataset remains the following steps, including tokenizing with nltk.word_tokenize, part-of-speech tagging with nltk.pos_tag, and lemmatizing with nltk.wordnet.WordNetLemmatizer.

Minor-6: More results regarding to PCA in Table2.

Generally, the variance explained drops smoothly. For the last layer of Vicuna, the first 300 principal components totally explain 22.28% of the total explained variances, where 300 is the number of components we studied in Section 4. The revised version will include the plot of the explained principal components.

Typos.

Thanks for pointing out all these typos. We will address them in our revised version. We will also make another detailed check on the whole paper.

Major-3: In places, it is difficult to follow the paper due to missing context.

Thanks for the suggestion, we will include more details in our revised paper, including but not limited to the following:

(1) We will discuss more observed phenomena in prior work that motivate this research in Section 2.

(2) We follow the discipline classification system in linguistics to discuss the "Linguistic levels". As mentioned in Section 4.2, the linguistic levels consider four levels of knowledge: phonology, morphology, syntax, and semantics. For more discussions of these linguistic disciplines, we would like to refer to the reference [1].

(3) We make the statement that "It's crucial to recognize that a word with a lower confidence doesn't necessarily imply it is a trivial word." in Section 3.1.1 with the following considerations. Formally, the next-word prediction process could be written as $p(y|x)$ where $x$ is the previous context and $y$ is a word. With Bayes equation, we have $p(y|x)\propto p(x|y)\cdot p(y)$. Regardless of the context $x$, the trivial words have a higher prior probability since they are much more common in the general corpus. Regarding the context $x$, $p(x|y)$is smaller for the $y$ are non-trivial words unless $(x,y)$ shows a common phase. Overall, a model is more confident if it tries to generate a trivial word, while it is less confident when it tries to predict a semantic word.

(4) We make the statement that "If we simply map ... it might not give us the full picture of all these patterns" in Section 4.1 with the following considerations. This Section aims to interpret the knowledge encoded in feedforward networks, and the most straightforward way is interpreting neurons (vectors of the weight matrix). However, many studies have pointed out that neurons are polysemantic [2, 3, 4, 5], leading to poor interpretability for humans. Therefore, we propose to decompose these neurons (patterns) via principal component analysis.

[1] James J Thomas. Illuminating the path:[the research and development agenda for visual analytics]. IEEE Computer Society, 2005.

[2] Trenton Bricken, et. al. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Anthropic Blog, 2023.

[3] Arora, Sanjeev, et al. "Linear algebraic structure of word senses, with applications to polysemy." Transactions of the Association for Computational Linguistics 6 (2018): 483-495.

[4] Elhage, Nelson, et al. "Toy models of superposition." arXiv preprint arXiv:2209.10652 (2022).

[5] Cunningham, Hoagy, et al. "Sparse autoencoders find highly interpretable features in language models." arXiv preprint arXiv:2309.08600 (2023).

Major-4: How will the developed XAI toolkit benefit future works?

We are planning to release the toolkit as a Python package (it could be pip installed). The toolkit will support various language models implemented by the Huggingface Transformers library well. The package will initially include these three proposed methods, but later, it will include other explanation methods.

The proposed three methods could be applied to various tasks and challenges. For example, the importance density could not only be used to evaluate the instruction following ability, but also other scenarios. Retrieval augmented generation (RAG) is one possible scenario, where they attach the retrieved documents as extra knowledge to the language models by assuming the models always utilize the information from the prompt. To verify this assumption during inference, we could compute the importance density over the retrieved documents.

Minor-2: It seems that Finding 3 is almost the same bottom-line finding as 1.

The main difference between Finding 1 and Finding 3 is in their perspective. Finding 1 is made based on the outcomes from the gradient-based attribution method over prompt-response pairs, while Finding 3 is drawn from a global explanation over model weights. Therefore, Finding 1 provides evidence that LLMs align with human preference by identifying instruction words, while Finding 3 is a deeper discussion of this phenomenon by indicating that LLMs acquire this ability by encoding more word-word patterns with instruction verbs.

Minor-3: Instruction-tuning or Instruction fine-tuning.

Yes, they are the same thing in this paper. The revised version will take the terms constantly.

Minor-4: Labelling details of Section 3.2.2.

The authors manually labeled a total of 632 prompt-response pairs for this experiment. Specifically, we consider the helpfulness of the response as the ability of instruction following. Therefore, if a response is helpful to the user, then we label it with “followed”. Please check Appendix B.1 for more details.