Towards Neuron Attributions in Multi-Modal Large Language Models

Dear Reviewer gq3m:

Thank you very much for your comments. We sincerely appreciate the time and effort you have dedicated to reviewing our work! Below, we meticulously provide responses to each of your comments and outline the modifications made to the manuscript based on your suggestions. Hope that our responses could address your concerns!

W1: The paper would benefit from being less notationally dense.

Thank you for your valuable feedback. Following your suggestions, we have revised our paper to improve clarity and readability. Specifically, we have streamlined the presentation by reducing notational density and simplifying definitions wherever possible (e.g., Equation 4~6).

Moving forward, we will be mindful of your suggestions and strive to maintain simplicity in both our descriptions and equations. We hope to contribute to curbing the trend of unnecessary complexity in academic writing.

Thank you again for your thoughtful comments!

Q1: I would like to see some discussion on whether the method is limited to attribution of concrete concepts.

Thank you for your insightful suggestion. To address your concern, we have expanded our discussion on the limitations of our method. Here are the key points:

• Firstly, our method relies on external segmentation models which can only segment concrete concepts. Hence, we acknowledge that this dependency makes our method challenging to attribute abstract concepts.
• Despite this limitation, we believe that this dependency (i.e., leveraging established external models from mature fields) can be beneficial, especially during the early stages of developing a new domain. Our experiments support this approach, demonstrating the advantages of cross-domain collaboration. Moreover, as segmentation models continue to advance, the upper bound of our method's capabilities will improve.
• Finally, we find it pertinent to mention our ongoing project that builds on this work and aims to attribute abstract concepts within multimodal LLMs. Specifically, it seeks to capture commonalities in the hidden layer when LLMs output abstract concepts, thereby circumventing the limitations of segmentation models. We plan to release it soon after this paper is accepted.

Furthermore, following your suggestion, we have added a subsection in the revised version, which elaborates on these points. We hope this addition enhances the rigor of the paper and provides readers with valuable insights.

Q2: The authors give citations to support that FFN predominantly encodes semantic knowledge. Is there some basic intuition for why this cannot be encoded in the self-attention block instead?

Thank you for raising this important question. The prevailing view — or basic intuition — suggests that the self-attention block is primarily responsible for gathering global features, while the FFN extracts contextual information based on these features, concretizing complex features into stable semantic concepts. Hence, the FFN potentially stores information that requires abstraction and synthesis beyond the immediate context.

Furthermore, from an experimental standpoint, several studies in recent years have supported this view. One of the most influential works is the ‘‘causal tracing’’ [1]. It employed neuron activation interventions and observed their impact on the output, achieving causal attribution of different blocks. We highly recommend reading this fascinating study for more detailed insights.

Finally, if you feel it would be beneficial, we are prepared to expand the Preliminary section in the revised version to include a more thorough discussion of both the theoretical and experimental evidence supporting this hypothesis.

Hope our response could address your concerns!

[1] Locating and Editing Factual Associations in GPT. NeurIPS 2022

Q3: It would be nice to have a comparison of computational costs, especially since the author claims that their neuron attribution method does not require time-consuming backwards-and-forward passes.

Thanks for your suggestion. Based on your feedback, we have added an analysis of computational costs in the revised version. The table below presents the average time (in seconds) required to attribute a concept. This demonstrates that avoiding the traditional backwards-and-forward pass significantly reduces computational overhead, effectively supporting our claims.

We hope these additional experiments provide a clearer understanding of the efficiency gains offered by our approach. Your input has been invaluable in enhancing the rigor and clarity of our paper. Thank you once again for your insightful feedback!

Q4: I wonder if this method would be applicable to audio modalities.

Thank you for your insightful question. Our method can indeed be applied to other modalities such as audio, since the new attribution function we propose is modality-agnostic and can generalize across different types of data.

To better illustrate this point, we conducted experiments with audio outputs using NExT-GPT. Here are the results:

Due to time constraints, we were only able to complete this single set. We hope for your understanding in this matter. Additional experiments are currently underway, and we plan to include them in the paper after acceptance.

Once again, we are deeply appreciative of the time and expertise you have shared with us. Your insightful suggestions have significantly enhanced our manuscript, and we are more than happy to add clarifications to address any additional recommendations and reviews from you!

Best,

Authors

Dear Reviewer VfTi:

Thank you very much for your comments. We sincerely appreciate the time and effort you have dedicated to reviewing our work! Below, we meticulously provide responses to your comments and outline the optimizations made to the manuscript based on your suggestions. Hope that our responses could address your concerns!

W1: How did you get the linear addition of the attention vector in Eq1?

Thanks for your question. As mentioned in Preliminary, vector $a$ in Eq 1 is the output of the attention block that has undergone normalization. Therefore, performing linear addition on $a$ is consistent with the structure of the transformer. Moreover, this linear addition is also supported by existing works on various downstream tasks, such as LLMs editing [1] and explanation [2]. Due to space limitations, we cannot elaborate further here, but we recommend reviewing these works for additional insights.

Additionally, following your feedback, we have enhanced the Preliminary in the revised version, particularly the descriptions related to Eq1. We believe this can strengthen the rigor and readability of the theoretical parts of our manuscript.

Hope our response could address your concern!

[1] Locating and Editing Factual Associations in GPT. NeurIPS 2022

[2] Multi-modal Neurons in Pretrained Text-Only Transformers. ICCV 2023

W2: The semantic relevance is deemed to emphasize what the proposed approach is doing, but might not be suitable for comparing with the baselines.

Thanks for your concern. We acknowledge that using semantic relevance for comparing different methods carries potential bias, as its theoretical foundation is similar to our approach, all based on residual connections.

Therefore, following your suggestion, we have revised the experimental section in the new version. Specifically, while other metrics (e.g., invariance and specificity) are retained, semantic relevance is only employed to validate the theoretical soundness of our method. We believe this adjustment will enhance the rigor of our manuscript.

We hope our response could address your concern!

W3: Presentation can be improved in Line 110, 209-210, 217-218, 260 and the definition of specificity.

We apologize for the oversight that resulted in some grammatical errors in the original text. Following your feedback, we have rechecked our paper and corrected grammatical errors and unclear expressions. Here are some examples of the corrections made:

• Line 110: To identify the contribution, we define these contribution scores as R.
• Line 209-210: The remaining results can be found in Appendix B.
• Line 217-218: For details on NAM and the baselines, please refer to Appendix A.
• Line 260：Figure 4 (a) presents the average invariance of concepts.
• Definition of specificity: Specificity is defined as the proportion of neurons that are crucial for a single concept solely.

We believe these revisions enhance the overall quality and readability of the manuscript. Thanks once again for your valuable comment!

Q1: Are the bars in Fig 3(c) stacked or overlayed? Is the green region T \cup I - T \cap I?

We apologize for any confusion caused. The bars in Fig 3 (c) are overlayed, and the green region represents T \cap I. To avoid this confusion, we have added this clarification to the reversion.

Q2: How do we know the activations of the neurons are signals. Why not lack of activations?

Thanks for your concern. Our work aims to explore which neurons play a critical role in generating multimodal content. Intuitively, neurons that are not activated do not seem to contribute to the output. Therefore, we naturally focus on the activated neurons solely.

However, as you mentioned, it is possible that non-activated neurons could contribute to the output in a subtle or indirect way. Hence, we have included this insight in the Discussion. In the future, we will endeavor to explore this point, and we look forward to contributing to this area.

We hope our response could address your query!

Q3: Related work should be moved to the main text. Is AcU the same as the residual stream in your approach?

Thanks for your suggestion. In response, we have moved the Related Work into the main text. Regarding the comparison with AcU, our method differs in that we consider both the activations and the residual stream. In contrast, AcU only considers the latter.

Q4: How do you know the neuron semantics and how they are correct? How do you know they are monosemantic? Why dog and shark are wrong for L28.U4786?

We apologize for any confusion caused. Here are our clarifications:

1. Neuron Semantics: As mentioned in Line 240-243, the unembedding matrix and the second linear of the FFN are treated as the projection from neurons to the vocabulary words. In this case, the word with the highest probability can be regarded as the neuron semantics. We cannot guarantee that these semantics are entirely accurate, but we consider it acceptable to use them as one of the references to assist in analyzing neuron characteristics.

2. Polysemantic: Many existing works have demonstrated the polysemantic nature of neurons. Our paper does not deny polysemy; instead, it aims to identify the most dominant of the multiple semantics. Thus, it does not conflict with polysemy but rather builds upon it.

3. L28.U4786: If L28.U4786 is selected as the explanation for "cat" while its semantics are "dog" and "shark", the explanation method is inaccurate. This is the point conveyed in Table 1.

Following your feedback, we have incorporated these clarifications into the appropriate sections to improve clarity and readability. Hope our response could address your concerns!

Once again, we are deeply appreciative of the time and expertise you have shared with us, and we are more than happy to add clarifications to address any additional recommendations and reviews from you!

Best,

Authors

Dear Reviewer VfTi,

Thank you very much for your active engagement and valuable feedback. We are pleased to hear that our previous response addressed most of your concerns, with the exception of Q4 (''How do your know the semantics of neurons？''). We appreciate the opportunity to provide further clarification. Since this year’s NeurIPS does not allow submitting a new version during rebuttal, we will provide a detailed explanation here to address your concern as thoroughly as possible within the limited space. We assure you that more detailed enhancements have been incorporated into the new version, and we sincerely hope for your understanding and trust in this process.

1. Residual Streams

To address your question, let’s first revisit the concept of residual streams. The output of the $l$-th layer in the LLM can be computed as follows:$\mathbf{h}^l=\mathbf{m}^l+\mathbf{h}^{l-1}+\mathbf{a}^l$

where $\mathbf{a}^l$ and $\mathbf{m}^l$ are the output of the attention and FFN block, respectively. Specifically, we have $\mathbf{m}^l=W_{out} \mathbf{r}^l$ where $\mathbf{r}^l$ is the vector of the intermediate layer in FFN and $W_{out}^l$ is the output embedding matrix in FFN.

Based on these, the output of the final layer $\mathbf{h}^{L}$ can be recursively derived as:$\mathbf{h}^L=\sum_{l=1}^L\mathbf{m}^l+\mathbf{h}^0+\sum_{l=1}^L\mathbf{a}^l$

It indicates that the output of any FFN module can contribute directly through linear addition to $\mathbf{h}^L$. This linearly accumulated flow of information is referred to as the residual stream.

Given the complexity of the non-linear relationships within LLMs which are difficult to intuitively understand and control, many downstream tasks (e.g., LLMs editing [1,2] and interpretability [3]) focus solely on the residual stream while reasonably neglecting other non-linear flows.

2. Neuron Contributions via Residual Streams

Revisiting the structure of LLMs, $\mathbf{h}^L$ is passed through the unembedding matrix $W_u$ to obtain a probability distribution over the vocabulary $\mathbf{y}$, with the word corresponding to the highest probability being the final output:$\mathbf{y}=W_u\mathbf{h}^L$

Since $\mathbf{m}^l$ contributes directly to the probability distribution through the residual stream, we define this contribution as $\mathbf{y}’$:$\mathbf{y}’=W_u\mathbf{m}^l=W_uW^l_{out}\mathbf{r}^l$

Here, $W_uW^l_{out}$ can be considered as a new unembedding matrix, and the probability distribution $\mathbf{y}’$ can be decomposed into contributions from each of the $k$ hidden neurons in $\mathbf{r}^l$:$\mathbf{y}’=W_u\sum_{i=1}^k W^l_{\text{out}i}r_i$

Thus, $W^{l}_{\text{out}i}{r}_i^l$ represents the distribution over the vocabulary for each hidden neuron. The word with the highest probability in this distribution is the one associated with that neuron. The intuition here is that regardless of the magnitude of a neuron’s output, its contribution will always be most inclined toward its associated word. For further details, please refer to [1].

3. Using These Semantics as Evaluation Metrics

As you pointed out, this method of deriving semantics could have limitations due to potential bias (since it only considers information within the residual stream). Therefore, as mentioned in our previous response, we did not use this as an evaluation metric in the reversion. However, given the success of focusing on residual stream in many fundamental LLM studies and downstream tasks [1,2,3], we believe these semantics still hold reference value. As such, we included this as an interesting experimental observation for visualizing neuron semantics, and we believe this visualization has the potential to offer readers valuable insights and perspectives on the behavior of neurons within MLLMs.

4. Other Evaluation Metrics

Although we removed this metric in the new version, the other metrics in our paper are unaffected by the residual stream and are therefore unbiased. The results from these metrics successfully validate the superiority of our method, ensuring the effectiveness of our experimental section.

Finally, we acknowledge that explainable MLLMs are still in their infancy, and there is currently no universally accepted metric. This work not only aims to provide a new paradigm but also actively explores usable, fair, and open-source evaluation metrics to advance this field. We humbly ask for your understanding and hope you can recognize the improvements we have made in the new version based on your feedback.

Thank you once again for participating in the discussion! We are ready to address any additional questions from you at any time and look forward to continuing our communication with you!

Best regards,

Authors

[1] Locating and Editing Factual Associations in GPT. NeurIPS 2022

[2] Mass-Editing Memory in a Transformer. ICLR 2023

[3] Multi-modal neurons in pretrained text-only transformers. ICCV 2023

Dear Reviewer VfTi,

We are extremely grateful for the insightful comments and the recognition you have shown toward our work. Your suggestion for optimizing the paper's presentation and experiments has significantly strengthened the rigor of our approach, making our contribution even more valuable to the community.

As our current score still hovering around the borderline, we humbly ask if you might consider the possibility of raising your score. Your support at this stage is immensely important to us! If you believe there are still areas in our paper that could be further improved, we would be more than happy to engage in any additional discussion to address any remaining concerns, particularly as the rebuttal period is about to conclude in just a few hours.

We are fully committed to advancing the field of explainable LLMs and contributing meaningfully to the community. Your feedback and support are invaluable to us in achieving this goal.

Thank you once again for your thoughtful engagement.

Warm regards,

Authors

Dear Reviewer VhwX:

Thank you very much for your comments. We sincerely appreciate the time and effort you have dedicated to reviewing our work! Below, we meticulously provide responses to each of your comments and outline the optimizations made to the manuscript based on your suggestions. Hope that our responses could address your concerns!

W1: The NAM method relies on external models (e.g., image segmentation models), which could limit the method's applicability and flexibility.

Thank you for raising this important concern. We acknowledge that reliance on external models may introduce limitations in terms of applicability and flexibility. However, we believe that this dependence is acceptable in the context of multimodal LLMs explainability for several reasons. Specifically,

1. Employment of External Models in Multimodal LLMs: Let's first consider the current landscape of multimodal LLMs (e.g., EMU [1], Unified-IO 2 [2], DreamLLM [3], RPG [4], CM3Leon [5], GILL [6], and NextGPT [7]). These models often incorporate external models (e.g., diffusion models) for multimodal understanding and generation. While these external models may introduce limitations in applicability and flexibility, they have not hindered the success and rapid development of multimodal LLMs. On the contrary, these mature models from well-established domains have become significant assets for multimodal research. Hence, we believe that the benefits of utilizing external models in this domain (i.e., multimodal research) far outweigh the potential limitations they might introduce.

2. Optimal Performance Across Diverse Task Configurations: According to our experimental results, our method achieves optimal performance across various modalities, datasets, and metrics. This impressive outcome, achieved through the integration of external models, demonstrates the substantial advantages of cross-domain collaboration. It indicates that the limitations introduced by external models (1) have a minimal impact on our method's application to mainstream downstream tasks, and (2) do not impede our model from achieving state-of-the-art results across various tasks. Moreover, another potential advantage of cross-domain collaboration is that as the external models continue to advance, the upper bounds of our method's capabilities will also improve, expanding the range of problems it can address. Therefore, we believe that cross-domain collaboration should not be constrained by the potential limitations introduced by external models.

3. Early Stage of Multimodal Explainability: The field of multimodal LLM Explainability is still in its early stages. Leveraging well-established external models to aid initial exploration is a common practice. We believe that as this field advances, the dependency on external models will gradually decrease, and domain-specific models will emerge to further propel research. We hope to contribute meaningfully toward achieving this goal.

Additionally, to address your concern, we have emphasized the potential limitations introduced by external models in the Limitations section of the revised version, and included the above points of discussion in the Appendix section, hoping to inspire further thought among researchers in the field of LLMs explainability.

Hope that our response and the revisions made to the manuscript could address your concern!

Once again, we are deeply appreciative of the time and expertise you have shared with us. Your insightful suggestions have significantly enhanced the rigor of our experimental design and enriched the content of our manuscript, and we are more than happy to add clarifications to address any additional recommendations and reviews from you!

Best,

Authors

[1] Generative Pretraining in Multimodality. ICLR 2024

[2] DreamLLM: Synergistic Multimodal Comprehension and Creation. ICLR 2024

[3] Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision Language Audio and Action. CVPR 2024

[4] Next-GPT: Any-to-any multimodal LLM. Arxiv 2023

[5] Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs. ICML 2024

[6] CM3Leon Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning. Arxiv 2024

[7] Generating Images With Multimodal Language Models. NeurIPS 2024

Dear Reviewer VhwX,

We are extremely grateful for the insightful comments you provided and for the recognition you have shown toward our work. Your suggestion to include additional discussion on the introduction of external modules has significantly strengthened the rigor of our approach, making our contribution even more valuable to the community.

As our current score still hovering around the borderline, we humbly ask if you might consider the possibility of raising your score. Your support at this stage is immensely important to us! If you believe there are still areas in our paper that could be further improved, we would be more than happy to engage in any additional discussion to address any remaining concerns, particularly as the rebuttal period is about to conclude in just a few hours.

We are fully committed to advancing the field of explainable LLMs and contributing meaningfully to the community. Your feedback and support are invaluable to us in achieving this goal.

Thank you once again for your thoughtful engagement.

Warm regards,

Authors

Dear Reviewer kLxz:

Thank you very much for your comments. We sincerely appreciate the time and effort you have dedicated to reviewing our work! Below, we meticulously provide responses to each of your comments and outline the modifications based on your suggestions. Hope that our responses could address your concerns!

W1. Testing on a broader range of models would strengthen the generalizability claims.

Thank you for your insightful suggestion. Following your feedback, in addition to the GILL and NextGPT, we conducted experiments on two additional multimodal LLMs, EMU and DreamLLM. The results, summarized in the table below, demonstrate that our explanation method achieves optimal performance across all metrics, thus further validating the generalizability of our approach.

We have incorporated these additional results into the revised version of the paper, along with a more detailed analysis. Due to the limited time available during the rebuttal phase, we were able to include only these two sets of experiments. We hope for your understanding in this matter.

Further experiments with additional multimodal LLMs are currently underway, and we plan to include them in the experimental section of the paper to enhance the robustness and comprehensiveness of our study. We hope that our response and the additional experiments meet your expectations.

Thank you once again for your valuable feedback!

W2. Exploring other modalities would provide a more comprehensive evaluation of NAM's capabilities.

Thanks for your valuable suggestion regarding the exploration of other modalities. Following your recommendation, we have extended our analysis beyond text and image modalities to include audio modality on NExT-GPT, which is capable of generating sound, and the following results validate its cross-modal generalizability.

Due to the time-consuming nature of generating other modalities, such as video, we were unable to complete all experiments within the short rebuttal period. However, these experiments are currently in progress, and we plan to include them in the updated version of the paper once completed. We appreciate your understanding on this matter.

Hope that our response and the additional experiments could meet your expectations!

W3. The method depends on external models (image segmentation models), which introduces a potential source of bias.

Thanks for raising this important concern. We acknowledge that our method depends on external models, which would introduce potential bias. However, we believe that this dependency is an acceptable aspect within the study of explainability in multimodal LLMs for several reasons.

1. External Models in Multimodal LLMs: Let's first consider the current landscape of multimodal LLMs (e.g., EMU, DreamLLM, Unified-IO 2, RPG, CM3Leon, and GILL). These models rely on external models (e.g., diffusion models) for multimodal generations. While these external models may introduce bias similar to the models our method employ, they have not hindered the remarkable success of multimodal LLMs. On the contrary, these well-developed models have become significant assets for multimodal research. Hence, we believe that the advantages of using external models far outweigh the potential bias introduced.

2. Optimal Performance Across Metrics: According to our experimental results, our method has achieved optimal performance across various metrics. This indicates that the additional bias introduced by external models has a minimal impact on the model's accuracy, and does not impede our method from achieving state-of-the-art results. Moving forward, as segmentation models advance, the upper bound of our method's capabilities will continue to improve.

3. Early Stage of Multimodal Explainability: The field of multimodal explainability is still in its early stages. Utilizing well-established external models from a mature field, such as image segmentation models, to aid initial exploration is inevitable. Furthermore, we believe that as the field advances, the dependency on external models will gradually decrease, and domain-specific models will emerge to further progress in research. We hope to contribute meaningfully toward achieving this goal.

To address your concern, we have added the above discussion into the revised version to enhance the rigor of the paper.

Hope our response could address your concern!

W4. The paper would benefit from ablation studies to isolate the impact of different components.

Thank you for your concern. Following your suggestion, we have included ablation studies in the revised version, which focuses on two main components: our proposed attribution score (AS) and the denoising module (DM). Here are the results:

These results demonstrate that each component of our design enhances the performance of our method. We hope these additional experiments could improve the overall integrity of our experiments.

Once again, we are deeply appreciative of the time and expertise you have shared with us. We are more than happy to add clarifications to address any additional recommendations and reviews from you!

Best,

Authors

Dear Reviewer kLxz,

We are extremely grateful for the insightful comments you provided and for the recognition you have shown toward our work. Your suggestion to include additional experiments has significantly strengthened the rigor of our approach, making our contribution even more valuable to the community.

As our current score still hovering around the borderline, we humbly ask if you might consider the possibility of raising your score. Your support at this stage is immensely important to us! If you believe there are still areas in our paper that could be further improved, we would be more than happy to engage in any additional discussion to address any remaining concerns, particularly as the rebuttal period is about to conclude in just a few hours.

We are fully committed to advancing the field of explainable LLMs and contributing meaningfully to the community. Your feedback and support are invaluable to us in achieving this goal.

Thank you once again for your thoughtful engagement.

Warm regards,

Authors