MINER: Mining the Underlying Pattern of Modality-Specific Neurons in Multimodal Large Language Models

1. A potential weakness of the paper lies in its approach to modality separation. The authors acknowledge the challenge of completely separating modalities due to the attention mechanism blending information across modalities. They hypothesize that information largely remains within its modality set as it passes through the LLM layers, with minimal transfer between modalities. They assume the attention mechanism functions only within distinct token sets, preventing information exchange between modalities. However, this assumption oversimplifies the attention mechanism, which is designed to capture dependencies regardless of modality. By limiting information flow, the authors may artificially constrain the model.

2. The authors restrict their analysis to neurons in the FFN module of the MLLMs, based on prior work highlighting the role of FFN in knowledge encoding. However, other modules, such as the attention layers, might also contain modality-specific neurons or contribute to modality-specific processing in ways not captured by this analysis.

3. The experiments primarily focus on image and text or audio and text combinations. Exploring more diverse combinations, such as image and audio or all three modalities together, could reveal a more complete picture of MSN interactions.

• The model should apply to modalities other than vision and audio, but the evaluation does not extend beyond these modalities. This is relatively minor, of course, since a paper that introduces a new method is not required to exhaust all empirical possibilities.
• The "modality separation" approach assumes minimal cross-modal information flow, which is a major oversimplification. It is not clear from an initial review to what extent removing this brick from the base of the theoretical structure collapses the rest. To a large extent this weakness is touched on in the first limitation in Sec 6.1, but it is not addressed. The natural interdependence between modalities, throughout the layers, should be better explained or explored. This is a more major limitation.
• It would be preferable to connect the concepts of ‘semantic telomeres’, for example, to real-world problems. What is the practical impact of this work?
• Other interpretability techniques for multimodal LLMs are not evaluated whatsoever.

• Validation of MSN Specificity: The evidence provided to confirm that the identified neurons are genuinely modality-specific (i.e., MSNs) is insufficient.
• Implications Remain Unclear: The practical implications and potential applications of identifying MSNs are not well-discussed, limiting the paper’s impact.
• Data Presentation and Clarity: Several figures, particularly Figure 6, are challenging to interpret due to their small size and lack of clear labels.

In my view, there are several weaknesses in the paper's approach:

1. Modality Separation Assumption: The assumption of modality separation, as defined in the paper, is overly generalized and lacks robustness.

• Dataset and Task Specificity: Modality separation can only be reasonably argued for certain datasets and tasks. For example, it might be valid for specific questions in Visual Question Answering (VQA) but is unlikely to hold across all question types. Moreover, it seems especially problematic to assume modality separation for captioning tasks and datasets, where contextual understanding of the entire image is essential.

• Dependence on Output Context: Another critical consideration is whether modality separation should be treated as a function of the output. In captioning tasks, the entire image often correlates with the output, as generating captions requires broader contextual information. In contrast, the input (or prompt) for caption generation may hold less significance, further challenging the assumption of modality separation. Could you, if possible, conduct an ablation study to assess the significance of the output modality in the importance calculation?

• Question on Importance Calculation: This brings up a key question regarding the calculation of neuron importance: is the importance score determined solely based on the input, or does it also consider the model's output? If not, would including the model’s output in this calculation lead to different importance scores?

2. Missing Definition: H_L is not defined in section 3.2. No reference is present in any of the equation 2 or 3 on the same page.

3. Dataset Size Impact: Table 2 illustrates that different datasets yield different neuron importance scores. However, it's not clear if varying the quantity of data within a single dataset also impacts these scores. For example, if using 100% of the dataset identifies important neurons $N = \{n_1, n_2, n_3, n_4\}$, would using only 50% or 33% of the dataset result in a different set of important neurons, or would it merely be a subset of $N$? If the importance neurons change, are they still importance neurons, or is there any underlying cause which we are not able to see.

4. Computation Requirements: The paper lacks an analysis of the computational resources needed to determine neuron importance. Since high-performance compute resources can be limited, an estimate of the required computational effort is essential to evaluate the feasibility of this explainability method.

1. The writing of this paper is very confusing, even with very low-level mistakes. It seems the authors do not understand what is important and the meaning of the formulation. For instance, in the lines 203 and 204, what does H mean? There is even no H in the equation. Besides, there are some inconsistent formats, such as fig. xx, and Figure xxx, which makes this paper not well-prepared for submission.
2. Since this paper focuses on FFN of MLLM, it is better to show the structure of the focused FFN, making it clearer to understand what the purpose of this paper is.
3. The analysis in the experiments is confusing. For example, in line 403 and 404, MMLU (0.31)? Also, in line 407, mentioned in Ob3? I do not understand what the paper means.
4. Whether the analysis in this paper depends on specific models or not? If so, is it accurate to claim that it is the characteristic of MLLM? I wonder if the performance of FFN in MLLM plays a similar role.