MODA: MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion Understanding

Response to Reviewer Ack2

Thank you for your insightful comments and questions. For your reference, we summarized the main results and included the attached file in our response to Reviewer XinM.

A1. Analysis for linear complexity of duplex attention alignment

(1) Complexity Analysis: As suggested, we provide the complexity analysis of the proposed duplex attention alignment. Given a multimodal token sequence $X\in\mathbb{R}^{N\times D}$, the computation cost comes from three operations, including Gram Matrix and alignment. In the Gram matrix, the computation cost is $O(D^2N)$. In the alignment, the computation cost is $O(DN)$, which in total is $O(D^2N+DN)=O(D^2N)$. Therefore, the proposed duplex attention alignment has a linear complexity in terms of token length, thus yielding a length extrapolation ability towards long multimodal context.

(2) Complexity comparison with other attention mechanism: In contrast, the baseline attention owns a nature of $O(DN^2)$ due to the similarity computation between each pair of tokens. Therefore, they have a higher computation complexity of $O(DN^2)$, where in most cases $N>>D$, especially for multimodal tokens, the number can be up to 128000 (e.g., max model len of Qwen2.5-VL).

A2. Ablation study on Attention mechanism

(1) As suggested, we further conduct a comparison between our MODA and other attention mechanisms, including baseline attention and SOTA attention mechanism. As shown below, MODA shows better performance among leading attention mechanisms due to its balance on multimodal tokens, while sparse and baseline attention may wrongly focus on the textual part.

(2) Besides, as shown in Tab.1 of the manuscript, we provided an ablation study on alternative designs from three aspects: how to align attention tokens (Tab.1b), how to fuse attention tokens (Tab.1c), and how to mask attention (Tab.1d).

A3. Computation cost: As suggested, we compare the computation cost of MODA and other attention mechanisms in terms of FLOPs, memory, and latency as below. We can observe that MODA achieves better performance on fine-grained understanding tasks (e.g., four vision-centric benchmarks) with a few increased computation costs.

Response to Reviewer Ack2

Thank you for your insightful comments and questions. For your reference, we summarized the main results and included the attached file in our response to Reviewer XinM.

A1&A2. Discussion on DDA

(1) Actually, DDA can be interpreted from the perspective of a multimodal token graph. The core of DDA, the attention mechanism, builds the linkage across multimodal tokens by computing the similarity of each pair of tokens [A,B]. The attention output is yielded by a weighted sum among tokens. Therefore, the computation can be seen as a densely connected direct graph through layer-by-layer attention, and the DDA problem can be reformulated as a mislinked graph problem, where the multimodal token flows to the textual part and misses the visual token from the input layer. DDA, therefore, decreases the interaction between visual and textual modality, and throws away 17% of important visual cues and is lost in the last embedding layer of MLLM. For the detailed diagram of multimodal token flow, please refer to Fig.1 in the attached file.

(2) Formula definition of DDA. Given the visual tokens $x_v^l$ and text tokens $x_t^l$ in the block $l$, the multimodal attention builds the link from two parts (i.e., self-modal $x_t^l \rightarrow x_t^{l+1}, x_v^l \rightarrow x_v^{l+1}$ and cross-modal $x_t^l \rightarrow x_v^{l+1}, x_v^l \rightarrow x_t^{l+1}$ ), where the links are commonly implemented by the pair-wise token similarity and weighted sum. However, the modality gap between tokens decrease the magnitude of links, as we observed, the link value of $x_v^l \rightarrow x_v^{l+1}$ and $x_t^l \rightarrow x_v^{l+1}$ decays exponentially with depth ($\alpha_{v \rightarrow v}^l \propto \gamma^l,\gamma \neq 1$). This misalignment propagates layer-wise, causing the cumulative error in cross-modal interaction to grow as $\mathbb{E}_L = \sum_l \gamma^l \epsilon_l$, where $\epsilon_l$ denotes the layer-specific alignment error. This phenomenon aligns with theoretical insights in [B], where pure attention mechanisms suffer from rank collapse – a critical factor exacerbating the attention distribution.

(3) MODA vs existing methods: To address this issue, we propose MODA, which introduces modality-aware gating and layer-wise error compensation to mitigate DDA. Unlike existing methods, MODA dynamically adjusts cross-modal interactions through modality-specific gates and compensates for alignment errors using adaptive propagation mechanisms. Further, MODA extracts the modality-specific feature and introduces the gram matrix as basis vectors for adaptive mapping. Thanks for the suggestions and we will re-organize the literature review in the introduction and related work.

[A] On the role of attention masks and layernorm in Transformers, NeurIPS, 2024

[B] Attention is not all you need: Pure attention loses rank doubly exponentially with depth, ICML, 2021

A3-1. Mapping function in V&T-aligner: The mapping functions in V-Aligner and T-Aligner are designed to address modality alignment issues and exploit four alternative designs as shown in Tab.1c of our manuscript. We explore direct replacement ($X_a$ as the original feature, $X_p$ as the mapped feature), concatenation, and element-wise addition for token fusion. The use of basis vectors enables structured modality space representation, improving alignment.

A3-2. Formulas: $||$ in Equ.5 represents the normalization operation.

A3-3. Self-attention&cross-attention: We modularize the influence of self- and cross-attention separately from two aspects: pull the tokens by alignment and correct the focus by masking.

A3-4. MODA for broader MLLMs: MODA can be integrated into the existing MLLMs by simply replacing each attention layer in the Transformer block. In our paper, we implement the most commonly used pre-norm type MLLM and verify its effectiveness on LLaMA-based MLLM. As a result, MODA can be integrated into the LLaVA series and Vicuna-based, Yi-based, Wizardlm2-based MLLMs because they share the same architecture as the implemented one of MODA. For clarity, we have prepared a torch-style pseudocode in Alg. 2 of the attached file.

A4-1. Detailed experimental setup: As suggested, we provide all the experimental settings, including the model, data, hyperparameter, and details. Due to the character limits, please refer to the Q1 of reviewer XinM.

A4-2&A4-3&A4-4. Analysis and Discussion of MODA: We thank the reviewer for the feedback. To address this, we added a detailed analysis of the advantages and disadvantages of each model and module across perception, cognition, and emotion tasks. MODA, as most MLLMs are, suffer from the risk of generating counterfactual responses, leading to false narratives or misrepresentations.

General Response

We sincerely appreciate all the Reviewers and the Area Chair for their time and effort in reviewing our paper. Following the valuable suggestions and insights provided in the reviews, we summarize the additional results and evidence included in the rebuttal based on the reviewers' suggestions:

• We provided the theoretical explanation of the deficit disorder attention problem and illustrated the diagram [A1&A2 for Ack2 and Fig.1 of the attached file].
• We conducted experiments on 9 benchmark datasets to discuss the negative impact of the critical issue of MLLM, i.e., the deficit disorder attention problem [A1 for Mf8z and Fig.2 of the attached file].
• We conducted more qualitative comparisons with SOTA MLLMs [Fig.3 of the attached file].
• We provided the pseudo-code for attention score computation and MODA for clarity [A6 for Mf8z, A3-4 for Ack2, and Alg. 1 & Alg. 2 of the attached file].
• We provided the computational complexity analysis of the proposed module and conducted comparisons among existing MLLMs and alternative attention mechanisms [A1 & A2 & A3 for cYUZ].
• We provided the detailed experimental setup and illustrated more information, including training time and hardware [A1 for XinM].
• We corrected the mentioned typos and carefully checked our manuscript again. Moreover, we also modified the figures and added the discussions as suggested [Fig.4 of the attached file].

Additionally, we follow the guideline of ICML 2025 and attach the figures as well as the pseudo-code in the anonymous link: https://anonymous.4open.science/r/icml25_rebuttal-F06A/README.md

---

Response to Reviewer XinM

Thank you for your insightful comments and questions.

A1. Training details, including backbone, data, hyperparameter, and hardware&time

(1) Same setting as Cambrian-1: To ensure a fair comparison, we adopt the same experiment setup, including the backbone and data, as Cambrian-1. Specifically, our training begins with a one-stage SFT following pretraining, which utilizes the pre-trained ViT and adaptor. During the training, we unfreeze the LLM backbone to fine-tune the attention part in the LLM and fully exploit the power of MODA.

(2) Details. We attach a configuration table below, which outlines the backbone, data, hyperparameter, and details during the training process. We will include details about training in the revised manuscript.

• Backbone: Cambrian-1 uses 4 vision encoders* including OpenAI CLIP ViT-L/14@336, SigLIP ViT-SO400M/14@384, DINOv2 ViT-L/14@518, and Open-CLIP ConvNeXt-XXL@1024. In contrast, we only use OpenAI CLIP ViT-L/14@336 as other popular MLLMs.
• Data: We follow the same setting as Cambrian and train our MODA by Cambrian-7M.
• Hyperparameter: We follow the common setting and train the model by using different lr for LLM and vision encoder.
• Hardware&time: For the 8B model, we use 2x A800 nodes to train for 6 days. For the 34B model, we use 4x A800 nodes to train for 14 days.

Response to Reviewer Mf8z

Thank you for your insightful comments and questions. For your reference, we summarized the main results and included the attached file in our response to Reviewer XinM.

A1. Claims on DDA

(1) Actually, DDA is supported by evidence from two aspects: rationale and observation.

• Rationale: Based on the graph theory, the interaction of multimodal tokens are discorrupted from the input level, where most important visual tokens are throw away. Further, the layer-by-layer propagation introduces the coefficient of accumulated ignorance. The illustration is included in Fig.1 of the attached file.
• Observation: First, attention scores exhibit a significant bias toward the textual modality, as shown in Fig.2a, where visual features are underrepresented. Second, Fig.2b&2c highlight a clear layer-wise attention decay, with attention inconsistencies becoming more pronounced in deeper layers. Finally, qualitative analyses in Fig.4&6 reveal that baseline models fail to capture subtle multimodal cues, resulting in incorrect or overly generic responses.

(2) We thank the reviewer for the valuable comments and will clarify this in the revised manuscript accordingly.

A2. Discussion on DDA

(1) Similar visual tokens lead to high attention scores. The authors would like to clarify that the statement of the reviewer 'adjacent tokens often exhibit higher similarity due to the spatial structure of images, naturally resulting in smaller self-attention scores.' is wrong. In fact, the similar visual tokens have a high attention score [A], according to the definition of attention $A = Softmax(QK/\sqrt{d})\propto QK$. $QK$ represents the similarity between tokens and is directly proportional to the attention score. Therefore, DDA suffers from imbalanced attention scores. The visual part is assigned a low attention score, leading to the neglect of important visual details.

(2) DDA brings negative impact in neglecting fine-grained details. We use a grounded MLLM (SA2VA) prompt by the ground truth answer to segment the key regions in visual content. Experimental results on 9 vision-centric perception, cognition, and emotion benchmarks revealed that the SOTA MLLM throws away 17% of the crucial visual tokens. For visualization results, please refer to Fig.2 of the attached file.

(3) DDA limits real-world applications. The imbalanced multimodal attention introduces bias in the flow of tokens, which can lead to inadequate token fusion and yield hallucination results [B]. As shown in Fig.1a &1b of our original manuscript, the model failed to recognize the waiter in the image background, posing critical challenges for practical scenarios, including OCR and conversational agent tasks.

[A] Attention is all you need, NIPS, 2017.

[B] Mitigating modality prior-induced hallucinations in multimodal large language models via deciphering attention causality, ICLR, 2025

A3. Comparison on fine-grained understanding: We would like to clarify that our manuscript already INCLUDES BOTH quantitative and qualitative direct comparisons with other leading MLLMs in fine-grained content understanding, as presented in Tab.2, 3, 4, and Fig.4, 7.

(1) Tab.2: We compare MODA with 6 close-sourced and 6 open-sourced MLLMs (GPT4V, Gemini 1.5 Pro, Grok-1.5, MM-1-30B, Mini-Gemini-HD, LLaVA-NeXT, Cambrian-1) on vision-centric and OCR tasks, which rely on fine-grained cue understanding.

(2) Tab.3&4: We compare MODA with 4 close-sourced and 7 open-sourced MLLMs (GPT4o, Gemini 1.5 Pro, Claude 3 Opus, Owen-VL-Max, Mini-Gemini-HD, LLaVA-NeXT, Cambrian-1, MMRole) on cognition and emotion tasks requiring contextual understanding.

(3) Fig.4&7: we provide qualitative comparisons with SOTA Cambrian-1.

(4) Fig.3 of the attached file: Additionally, we conduct further quantitative comparisons with Cambrian-1 on human-centric understanding and planning tasks.

A4. Framework and pipeline in Fig.3 and Sec. 3: Thanks for the reminder and we include the modified pipeline and caption as Fig.4 of the attached file. Besides, we will add an overview section in Sec 3.3.

A5. Illustration of Fig.2: The x-axis and y-axis represent the attention values and the corresponding probability. The x-axis ranges from 1e-4 to 1e-1 and is a logarithmic scale.

A6. Attention score: Following [C], we compute token-level attention scores on all the testing set and perform average. The pseudocode is included as Alg.1 in the attached file.

[C] Efficient streaming language models with attention sinks, NeurIPS, 2024

A7&A8. Writing: (1) [M] replaces modular masked attention with learnable tokens to control attention distribution. (2) The term refers to the residual features before Gram matrix mapping. (3) For the token sequence $X^m \in \mathbb{R}^{N_m \times D}$ of modality $m$, the mask controls its visibility with the entire multimodal sequence X \in \mathbb{R}^{(N_m+N_\bar{m}) \times D}.