Coupled Mamba: Enhanced Multimodal Fusion with Coupled State Space Model

Thank you for your detailed and insightful assessment of our paper; we are deeply grateful for the time and expertise you dedicated.

W1: Parallelism through global convolution:

S4 relies on a global convolution kernel to achieve parallelism, while Mamba iterates on Equation 3 to compute intermediate results, ultimately accelerating computation using a global convolution kernel. In our Coupled Mamba, we retain the parallel scanning scheme within each modality. During the parallel scanning process, we iteratively combine intermediate results from each modality, integrating historical states across modalities. Simultaneously, we derive a multimodal global convolution kernel to fuse results from intra-modal parallel scanning. This design ensures that computations for each convolution kernel and input subregion remain independent, facilitating parallel processing of different convolution kernels or input blocks.

W2: Summation is too simple to model complex modality interaction

Our method utilizes the state transition matrix $S_m$ to evolve the dynamic process of multimodal fusion. While summing up historical states might seem straightforward, our model effectively filters historical information from different modalities through the coupling matrix $G$, retaining only the filtered, crucial information. Additionally, we introduce self-learning scalars that enable the model to adaptively learn the contribution of each modality to the task. Numerous experiments have demonstrated that, compared to other widely used fusion schemes such as averaging and cascading, our summation scheme not only enhances speed but also delivers superior performance.

W3: Comparison with other multi-modal Mamba models

Thank you for suggesting the concurrent multimodal Mamba work, including VL-Mamba and Cobra. Both VL-Mamba and Cobra are based on large language models (LLM) and are designed for VQA tasks. Our Coupled Mamba introduces Mamba to general multimodal tasks instead. In addition, VL-Mamba uses VSS Block to extract features from visual input, and then aligns textual features with visual features for fusion and inputs them into LMM for training. Cobra uses two visual models, DINOv2 and SigLIP, to extract visual features, and then aligns visual features with textual features for multimodal fusion. The Mamba models in both VL-Mamba and Cobra serve merely as feature extractors, and the multi-modal feature fusion is conducted within a LLM. They do not present any new schemes for multi-modal fusion. We will add both qualitative and quantitative comparisons with other concurrent multimodal Mamba works, including the proposed VL-Mamba and Cobra in the revision.

Q1: meaning of 'average (concat) fusion'

Yes, 'average fusion' directly averages multimodal features before processing them with a single Mamba, while 'concat fusion' concatenates multimodal features. We will make this point clearer.

Thank you for your detailed and insightful comments and feedback, and we are deeply grateful for the time and expertise you dedicated to reviewing our paper..

W1: Incomplete and inconsistent references

We thank the reviewer for pointing out this issue. We will carefully proofread and correct all incorrect references, formatting issues, and typos.

W2: The form of $S_0$ is the same with SSM

In our Coupled Mamba, $S_0$ is obtained by multiplying the input-related $\overline{A}$ and the devised coupling matrix $G$. Please refer to the detailed proof in the appendix. It can be interpreted as a shared state transition matrix that transfers the coupling states based on a certain probability derived from the comprehensive state at time $t-1$. We use this same form for consistency with SSMs for unification and better interpretability.

W3: The relationship between Eqn. 5 and the probability transition matrix of CHMM

Thank you for the valuable feedback. CHMM has been widely adopted for multi-modal fusion [1, 2], utilizing the probability transition matrix to integrate information from multiple modalities. Our design of Equation 5 is inspired by CHMM, as the state propagation sequences in SSM resemble the state chains in CHMM. Equation 5 can also be interpreted as the probability transition between continuously evolving states, analogous to the transition of discrete states in CHMM.

W4: Eqn. 6 sets weights of hidden states from all modalities to 1

In our implementation, we use a learnable scalar to weight each modality in Eqn. 6. This has not been reflected in the paper, and we will clarify this point in the revision. Additionally, in response to your suggestion, we conducted a comparative experiment by setting the weights of all historical states to 1. The results are shown in Table 5 (in the PDF). By incorporating a learnable scalar to weight each modality in the equation, we address the issue of varying contributions from different modalities to the task, thereby improving the performance of multimodal fusion.

W5: Inconsistent dimensions of $s_0$ and $s_m$: Thanks for pointing this issue out. Sorry for the typos. The correct dimensions for $S_{m}$ are [B,L,E,]. We will correct in the revision.

W6: Lack corresponding inference time, FLOPs, or parameter data in Table 1 We apologize for the missing information. In addition to providing only the inference time, FLOPs, or parameter data, we further compared our Coupled Mamba with several SOTA methods on inference time, FLOPS, and parameters. The results are shown in Fig. 2, Fig. 1, and Tab. 3 of the rebuttal PDF. Additionally, we also compared memory usage in Fig. 3. Our method exhibits the lowest memory consumption and fastest inference speed.

W7: In the Introduction, there is an extra space in 'Transformer-based

Thank you for your careful review. We will make revisions in future versions.

Q1: Please refer to the reponse to W2.

Q2: Please refer to the reponse to W3.

Q3: $\bar{A}_{m,m}$ should differ from $\bar{A}$.

Thank you for your suggestion and feedback. Note that all $\bar{A}{i,m}$ $, i \in 1 \ldots M$ in Eqn. 5 are learned parameters during network training. To preserve the generalizability of the model, we do not specifically emphasize the intra-modal transition $\bar{A}_{m,m}$, and instead let the network learn to prioritize modalities.

Q4: Ablation experiment on simplification from Eqn. 5 to Eqn. 6

Thank you for the valuable suggestion. We have conducted ablation study follow your suggestion, as illustrated in Table 4 in the rebuttal PDF. The results indicate that this simplification reduces both the number of parameters and memory usage, and also improves inference speed, with minimal impact to the performance.

[1]Garg A, Naphade M, Huang T S. Modeling video using input/output Markov models with application to multi-modal event detection[J]. Handbook of Video Databases: Design and Applications, 2003: 23-44.

[2]ZHANG Yu-zhen,DING Si-jie,WANG Jian-yu,DAI Yue-wei,CHEN Qian.Event Detection by Fusing Multimodal Objects Using HMM[J].Journal of System Simulation,2012,24(8):1638-1642.

Thank you for your detailed and insightful comments and feedback, and we are deeply grateful for the time and expertise you dedicated to reviewing our paper..

W1: Provide insights over combining Mamba branches for multi-modal fusion

Effective multi-modal fusion hinges on balancing inter-modality information exchange while preserving independent intra-modality learning [1,2]. Our approach, Coupled Mamba, achieves this by integrating historical information across modalities while maintaining intra-modal state autonomy. Specifically, Coupled Mamba fuses the state of each modality's chain with adjacent chains from previous time steps across modalities. This integration ensures that each modality's current memory incorporates crucial historical context from multimodal data, progressively building a comprehensive model over time. We will make this point more clear.

W2: Lack analysis or interpretability of the model's multi-modal fusion mechanism

Thanks for the insightful feedback! In our Coupled Mamba, we use a separate Mamba branch for each modality, and fuses the state of each modality's chain with adjacent chains from previous time steps using a learnable state transition matrix ($A_{i, m}$ in Eqn. 5 and $S_{m}$ in Eqn. 6) to ehance multi-modality fusion. The state transition matrix selectively integrates important historical information from other modalities, while neglects information that's insignificant. By utilizing the coupling transition between states, Coupled Mamba is tightly connected in time steps, effectively modeling multimodal data.

This mechanism is fundamentally different from existing multimodal fusion schemes, where existing fusion methods can be categorized into

1. attention mechanisms. this method achieves multimodal information fusion through cross attention mechanisms.
2. the encoder decoder method. This method extracts high-level features of different modalities through an encoder, maps them to a low dimensional space, and then uses a decoder to generate predictions from these latent representations.
3. Graph neural network method. This method models multimodal data by constructing graph structured data.

These designs all focus on learning good multi-modality features. In constrast, Coupled Mamba ensures intra-modal indenpence (as we use separate Mamba branches for each modality), and fuse important information from multi-modalities using the learnable state transition scheme, which emphasize more on the information fusion process. We believe that's the reason why our approach outperforms across all multi-modal tasks in experiments.

We will explain this point clearer in the revision.

W3: The speed and linear complexity advantage is from Mamba

We only partially inherit the speed and linear complexity advantage of the original Mamba, by preserving the parallel scan scheme within each modality. However, the original Mamba and other state-space models were originally designed to process unimodal data and cannot handle cross-modal fusion well. When naively adapted for multi-modal data, as indicated by the 'concat' and 'mamba' fusion approach shown in Table 9 in the main paper, they require higher memory and computational resources, resulting in inferior performance compared to our coupled design.

W4: Experiments are limited to sentiment analysis tasks

Thanks for this helpful suggestion! We conducted additional experiments on the BRCA benchmark and MM-IMDB benchmark, and the results are presented in Table 1,2 in our rebuttal PDF. The BRCA benchmark focuses on predicting the PAM50 subtype classification of breast cancer, based on three complex data modalities, i.e., the mRNA expression, DNA methylation and miRNA. while the MM-IMDB dataset is used for movie genre classification from movie intro, images. Despite the differences in data modalities and tasks, our Coupled Mamba achieves the best performance on both benchmarks compared to existing SOTA methods, demonstrating its generalizability.

Q1: Meaning of "mamba fusion" in Table 9

"mamba fusion" means establishing a seperate Mamba branch for each modality. Subsequently, the output features of these Mamba branches are merged through a weighted aggregation mechanism, consistent with the late-fusion paradigm embodied by methods such as LMF and Mult. We will further elaborate on this process in the revision.

Q2: Provide more interpretability of combining Mamba structures:

Please refer to our response to W1

[1] Wang, Yikai, Wenbing Huang, Fuchun Sun, Tingyang Xu, Rong Yu, and Junzhou Huang. 2020. "Deep Multimodal Fusion by Channel Exchanging." NeurIPS 2020.
[2] Li, Yaowei, Ruijie Quan, Linchao Zhu, and Yi Yang. n.d. "Efficient Multimodal Fusion via Interactive Prompting." CVPR 2023

Thank you for your detailed and insightful comments and feedback, and we are deeply grateful for the time and expertise you dedicated to reviewing our paper..

W1: The improvement in the regression task on CMU-MOSEI seems marginal

The CMU-MOSEI dataset mostly consists of short sequences: most of the videos, audios, and texts are only 1-5 seconds in duration. Mamba is more suitable for modeling relatively long sequences, which results in marginal improvement of Coupled Mamba on the CMU-MOSEI dataset. In contrast, for datasets with longer sequences, such as data in CH-SIMS, CH-SIMSV2, BRCA, and MM-IMDB datasets, our Coupled Mamba can effectively enhance multi-modal fusion and outperforms the state-of-the-arts by a relatively large margin. We will include a discussion on the impact of sequence lengths in the limitations section.

W2: Less experiments on aligned data and evidences of robustness:

The fewer experiments on 'aligned' data are influenced by the composition of the datasets used, where there is a predominance of unaligned data. This is reasonable, as aligned data is more difficult to collect. Moreover, many existing classification methods for the CMU-MOSEI dataset primarily focus on unaligned data; hence, we use unaligned data for a fair comparison. Besides, experiments on aligned data in the main paper Table 1 have demonstrated our excellent performance for aligned data input. Furthermore, unaligned data is more challenging to process, and our approach achieves superior performance regardless of aligned or unaligned data.

For robustness, we have conducted experiments on aligned/unaligned data (Table 1 of the main paper), various modalities (text, audio, video, mRNA, miRNA, DNA, as shown in Tables 1 and 2 in the rebuttal PDF), sequence lengths (ranging from 100 to 500), and text in different language domains (English vs. Chinese). All experiments show that our Coupled Mamba achieves superior performance, demonstrating its robustness across various data input.

Thank you for your thorough and insightful feedback. We genuinely appreciate the effort and expertise you invested in reviewing our paper.

W1: Lack of speed and memory analysis

We thank the reviewer for this insightful feedback. In our rebuttal PDF, we have included a comparison of memory usage(Fig 3), inference speed(Fig 2), flops(Fig 1), and parameter size (Table 3) with Mult[1], MISA[2], TFN[3], and LMF[4], all using inputs directly instead of hidden states. Additionally, Tab. 3 presents a detailed comparison of parameters. From both Fig. 1 and Tab. 3, it is evident that our Coupled Mamba consumes significantly less memory and achieves the fastest inference speed for sequences from short to long.

W2: Expriments on more multi-modal data and tasks to demonstrate generalizability

Thanks for this helpful suggestion! We conducted additional experiments on the BRCA benchmark and MM-IMDB benchmark, and the results are presented in Tab 1,2 in our rebuttal PDF. The BRCA benchmark focuses on predicting the PAM50 subtype classification of breast cancer from mRNA, DNA, miRNA data, while the MM-IMDB dataset is used for movie genre classification from text and images. Despite the differences in data modalities and tasks, our Coupled Mamba achieves the best performance on both benchmarks compared to existing SOTA methods, demonstrating its generalizability.

Q1: Reason of converting to the convolutional representation for parallelization

The parallel scanning scheme in Mamba computes independent partial states recurrently from inputs and then combines these partial states using a global convolution in a hierarchical manner to achieve parallelism. In our Coupled Mamba, we adapt this approach by conducting recurrently computation of partial states within each modality. We enhance this process with a summation operation through a global convolutional kernel, enabling multi-modal fusion capabilities while preserving the inherent advantages of parallelism. To be more specific, the parallel scanning process generates intermediate results, which we fuse using a summation scheme with a global convolutional kernel across modalities. This scheme is inherently parallelizable and accelerates the fusion of multimodal data. We will clarify this point further in the revision.

Q2: Explanation of average, concat, and mamba fusions in the ablation study

average fusion involves directly averaging the multi-modal features. concat fusion concatenates the multi-modal features. The fused feature is then fed into a Mamba model, followed by a pooling and head layer for the final output. For Mamba fusion, we create a separate Mamba branch for each modality. The output features of each branch are then combined using a weighted summation (following late fusion schemes as in LMF, Mult). We will clearify this in revision.

Q3: The meaning of unaligned and aligned in experiments

aligned data refers to data from different modalities that are precisely synchronized in both spatial and temporal dimensions, such as synchronized audio and video. In contrast, unaligned data refers to instances where different modalities are not synchronized, presenting a more challenging scenario. Our method demonstrates superior performance on both aligned and unaligned data.

Q4: Some details about the dataset are missing

Our experiments utilize datasets comprising data from video, audio, and text modalities. The CMU-MOSEI dataset includes both aligned and unaligned data, with sequence lengths of 50 for aligned text, audio, and video, and 50, 500, and 375 for unaligned text, audio, and video, respectively. The CH-SIMS and CH-SIMSV2 datasets contain exclusively unaligned data, with sequence lengths of 39, 400, and 55 for text, audio, and video in CH-SIMS, and 50, 925, and 232 in CH-SIMSV2.

[1] Tsai, Yao-Hung Hubert, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. “Multimodal Transformer for Unaligned Multimodal Language Sequences.” In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. doi:10.18653/v1/p19-1656.

[2] Hazarika, Devamanyu, Roger Zimmermann, and Soujanya Poria. 2020. “MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis.” Cornell University - arXiv,Cornell University - arXiv, May.

[3] Zadeh, Amir, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. “Tensor Fusion Network for Multimodal Sentiment Analysis.” arXiv: Computation and Language,arXiv: Computation and Language, July.

[4] Liu, Zhun, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, and Louis-Philippe Morency. 2018. “Efficient Low-Rank Multimodal Fusion with Modality-Specific Factors.” In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). doi:10.18653/v1/p18-1209.