Towards Holistic Multimodal Interaction: An Information-Theoretic Perspective

Question 1. About real-world experiments:

Question 1a. lack extensive empirical analysis.

Thank you for your valuable comment. At present, measuring interactions under real data conditions remains imprecise. This challenge is largely due to the lack of an accurate understanding of interactions of each sample, which remains an open problem in the field. To address this, we have taken two approaches. First, we expanded our experimental scope significantly to demonstrate the versatility of our method across various contexts, which is detailed in Appendix B.3. The validation experiments including extensive modalities, tasks (Appendix B.3.2), and backbones (Appendix B.3.5), three modalities (Appendix B.3.3), with temporal dynamics (Appendix B.3.4) and more analysis about ablation (Appendix B.3.6). Second, we conducted validation experiments on synthetic datasets with controllable interactions for each sample to more clearly illustrate the learning mechanisms of our method. Moving forward, with more accurate measurements of multimodal interactions to be investigated, we expect to gain a deeper understanding of our approach.

Question 1b. Marginal improvement over some datasets.

Thank you for raising this point. These results were observed in the AV-MNIST and CMU-MOSEI (V+T) task, which is inherently challenging. AV-MNIST, a synthetic dataset derived from MNIST, has traditionally seen only modest improvements with previous methods compared to joint training. For example, the best performance in prior work is 72.8%, compared to 71.7% for joint training [1]. In our experiments, our method achieved 73.5%, significantly surpassing the joint training benchmark of 72.6%, thereby validating the efficacy of our approach. Additionally, we have replaced this synthetic dataset with real-world data from the UCF-101 dataset, which includes Optical Flow and RGB modalities, as detailed in Table 2. We present partial results here, demonstrating the effectiveness of our method across real-world datasets.

Sentiment analysis tasks are particularly challenging on the CMU-MOSEI dataset, where even marginal improvements are valuable [2]. Previous studies have reported a maximum improvement of 1.5% over joint learning (LF-Transformer) in binary classification tasks across three modalities [1]. In this paper, we tackle a more challenging setup with only two modalities, making the task even more difficult. Our method achieves a 1.2% increase in the (A+T) setup for a three-way classification task. Moreover, our method demonstrates improvement in the (V+T) modality, whereas some comparison methods experience a drop in performance. This highlights both the complexity of learning in this setup and the effectiveness of our proposed approach.

Question 1c. Dataset, benchmarks are also limited in terms of scope.

Thank you for your suggestion. Existing datasets typically include Audio, Visual, and Text modalities, with sample sizes ranging from 5,000 to 30,000. Following your suggestion, we have expanded our experiments to include more modalities and larger-scale datasets. In the revised manuscript, we validate our methods across various datasets and combinations of modalities: Audio + Text on the UR-FUNNY dataset for humor detection [3], RGB + Optical Flow on the UCF101 dataset for action recognition[4], mRNA + methylation data on the ROSMAP dataset for Alzheimer's Disease diagnosis [5], and Audio + Visual on the VGGsound dataset for audio recognition, which includes an extensive sample size of 200,000 [6]. Detailed results for these diverse modalities are presented in Appendix B.3. Experimental outcomes confirm that our method significantly outperforms the baseline across different scales and modalities.

Question 4. Does overlap lie in DMI assumption?

Thank you for this invaluable question. To illustrate this point, let us consider the use of a smiley face in sarcasm detection within the visual modality. The interpretation of the smiley face as conveying unique information is highly context-dependent. For example, if every data point includes a smiley face, there is no discernible correlation between its presence and sarcasm—actually, the smiley face conveys no information about sarcasm in this scenario. Conversely, if the smiley face appears exclusively under specific conditions, such as during moments of happiness or sarcasm, it then strongly correlates with sarcasm, thus illustrating the unique interaction our method aims to capture.

Our approach is designed to decompose each data point into multiple types of interactions. In the scenario described, the uniqueness interaction captures task-related correlations, while the synergy interaction identifies emergent correlations.

We have also applied our methodology to the humor detection dataset UR-FUNNY, which is known to contain an amount of synergy information [7]. The experimental results demonstrate that our method can effectively capture this synergy information, leading to improved performance.

[1] P. P. Liang, Y. Lyu, X. Fan, Z. Wu, Y. Cheng, J. Wu, L. Chen, P. Wu, M. A. Lee, Y. Zhu et al., “Multibench: Multiscale benchmarks for multimodal representation learning,” arXiv preprint arXiv:2107.07502, 2021.

[2] H. Wang, S. Luo, G. Hu, and J. Zhang, “Gradient-guided modality decoupling for missing-modality robustness,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 14, 2024, pp. 15 483– 15 491.

[3] M. K. Hasan, W. Rahman, A. Zadeh, J. Zhong, M. I. Tanveer, L.-P. Morency et al., “Ur-funny: A multimodal language dataset for understanding humor,” arXiv preprint arXiv:1904.06618, 2019.

[4] K. Soomro, “Ucf101: A dataset of 101 human actions classes from videos in the wild,” arXiv preprint arXiv:1212.0402, 2012.

[5] P. L. De Jager, Y. Ma, C. McCabe, J. Xu, B. N. Vardarajan, D. Felsky, H.-U. Klein, C. C. White, M. A. Peters, B. Lodgson et al., “A multi-omic atlas of the human frontal cortex for aging and alzheimer’s disease research,” Scientific data, vol. 5, no. 1, pp. 1–13, 2018.

[6] H. Chen, W. Xie, A. Vedaldi, and A. Zisserman, “Vggsound: A large-scale audio-visual dataset,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 721–725.

[7] P. P. Liang, Y. Cheng, X. Fan, C. K. Ling, S. Nie, R. Chen, Z. Deng, F. Mahmood, R. Salakhutdinov, and L.-P. Morency, “Quantifying & modeling multimodal interactions: An information decomposition framework,” in Advances in Neural Information Processing Systems, 2023.

Question 2. About synthetic experiment:

Question 2a. The statistical relationships of three types of interactions

Thank you for highlighting this point. For synthetic data, we specifically construct samples to correspond to particular types of interactions. Each sample exclusively contains one type of interaction, and for simplicity, different interactions are encoded in distinct dimensions of the inputs. When a sample is determined by specific interaction, dimensions corresponding to other interactions will be set into noise. This approach ensures that each sample uniquely represents one type of interaction. We provide further clarification of this setting in Appendix B.4.

Question 2b. Interaction generation for $\frac{1}{4} U + \frac{3}{4} R$.

Thank you for raising this point. In Figure 1, two types of interactions are depicted: redundancy and uniqueness. The notation $\frac{1}{4} U + \frac{3}{4} R$ indicates that $\frac{1}{4}$ of the data is sampled from unique interactions, while $\frac{3}{4}$ is sampled from redundant interactions. Each sample corresponds with a certain type of interaction. We clarify this description in the revised manuscript, Appendix B.4.

Question 2c. Synthetic data and DMI architecture.

Thank you for your question. Here, we use 10000 samples in the synthetic dataset, and the DMI architecture is similar to the real-world experiment (detailed in Appendix B.2), with the backbone changing into a 3-layer neural network with ReLU as an activate function.

Question 2d. More boolean mixture of interactions.

Thank you for pointing this out. In response, we have incorporated additional types of interactions, including combinations of (OR + XOR) and (AND + OR + XOR), to assess the effectiveness of our method in handling more complex interactions. The detailed results indicate that the DMI approach can effectively capture these complex interactions, even when multiple Boolean variables are involved. These results are presented below and are also summarized in Table 4 of the revised manuscript.

Question 3. Why synergistic information are considered task-irrelevant features?

Thank you for addressing this important aspect. Task-irrelevant information is defined as information within unimodality that is not directly related to the task at hand. According to the definition of synergy data (refer to Equation 11 and [1]), synergy arises when individual modalities alone provide no information for task completion, yet their integration generates emergent information that is crucial for the task. This emergent information, which we term synergy, arises indirectly from unimodality. Consequently, the information derived from synergy interaction belongs to the task-irrelevant category within each unimodality. Thus, we characterize the information emergent from the combination of two task-irrelevant features as the learned synergy. We clarify this distinction in Section 3.4.

Dear Reviewer 9UkZ,

Thank you for reviewing our work and for providing constructive suggestions. We have carefully considered your comments and made the necessary adjustments. We would be thankful if you could inform us whether the revisions adequately address your concerns. If further clarification is needed, we are happy to provide additional details.

We greatly appreciate your positive feedback and are pleased to hear that your questions have been addressed.

Thank you again for your valuable insights. If you require any further clarification or additional experiments, please feel free to reach out to us.

Question 2. Questions with the experiments:

Question 2a. Limited improvements on AV-MNIST and CMU-MOSEI (V+T).

Thank you for pointing this out. The AV-MNIST and CMU-MOSEI (V+T) tasks are inherently challenging to learn. AV-MNIST is a synthetic dataset derived from MNIST, which has shown only modest improvements with previous methods compared to joint training. For example, prior work reports the best performance at 72.8%, while joint training (LF) achieves 71.7% [3]. In our experiments, our method surpasses joint training by 0.9%, validating the effectiveness of our approach. Furthermore, we replace this synthetic dataset with the real-world dataset, UCF-101, which includes Optical Flow (OF) and RGB modalities, as detailed in Table 2. Below, we present partial results, demonstrating the effectiveness of our method across real-world datasets:

Besides, sentiment analysis on the CMU-MOSEI dataset is also challenging. Previous studies have reported a significant improvement of 1.5% over joint learning in binary classification tasks across three modalities [3]. In this paper, we consider a tougher task with only two modalities. Our method achieves a 1.2% increase in the (A+T) setup for a three-way classification task, and demonstrates improvement in the (V+T) modality, whereas some comparison methods experience a drop in performance. This highlights both the complexity of learning in this setup and the effectiveness of our proposed approach.

Question 2b. Modifications specific to each modality in ResNet18.

Thank you for your question. The modification involves adapting the network to accommodate modalities that differ from the typical three-channel RGB input. Specifically, for the audio modality, which has a single channel, and the optical flow modality, which consists of two channels, we modify the channel dimension of the first layer of ResNet18 to correspond with the channel dimensions of these modalities.

Question 2c. Details about MOSEI task.

Thank you for the inquiry. The task conducted on MOSEI is a sentiment analysis task. These samples are divided into positive, negative, and neutral, following the setting in the previous study [4].

Question 2d. The performance of the unimodal method is missing.

Thank you for your suggestion. We have supplemented the unimodal performance in Table 2 and Table 3 in the Experiment section, highlighting the improvements achieved by the multimodal methods.

Question 2e. Why DMI-TC and DMI-CD worse than DMI-FC.

Thank you for your valuable comment. These three methods are based on our proposed DMI architecture. For DMI-FC, we adopt the principle of decomposition but replace the variational decomposition with a fully-connected layer. This can degrade the decouplement among modalities. DMI-TC and DMI-CD, on the other hand, represent partial decompositions within the DMI framework. The superior performance of DMI-FC showcases the significance of comprehensive decomposition. But with the decoupling among modalities, DMI can achieve better decomposition and thus achieve better performance.

[1] P. P. Liang, Y. Cheng, X. Fan, C. K. Ling, S. Nie, R. Chen, Z. Deng, F. Mahmood, R. Salakhutdinov, and L.-P. Morency, “Quantifying & modeling multimodal interactions: An information decomposition framework,” in Advances in Neural Information Processing Systems, 2023.

[2] A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy, “Deep variational information bottleneck,” arXiv preprint arXiv:1612.00410, 2016.

[3] P. P. Liang, Y. Lyu, X. Fan, Z. Wu, Y. Cheng, J. Wu, L. Chen, P. Wu, M. A. Lee, Y. Zhu et al., “Multibench: Multiscale benchmarks for multimodal representation learning,” arXiv preprint arXiv:2107.07502, 2021.

[4] C. Hua, Q. Xu, S. Bao, Z. Yang, and Q. Huang, “Reconboost: Boosting can achieve modality reconcilement,” arXiv preprint arXiv:2405.09321, 2024.

Question 1. Question with the method section.

Thank you for your valuable comment. Here, we provide a detailed explanation of questions and modify the paper to describe our method detailedly.

Question 1a. Why does synergy only contain task-irrelevant information?

Thank you for addressing this important aspect. Task-irrelevant information is defined as information within unimodality that is not directly related to the task at hand. For example, in XOR data with two independent binary variables, there is no correlation between the target and either modality. However, when two variables are not independent (e.g., $x^1 = x^2$), task-related information occurs, meanwhile additional interactions beyond synergy are observed. Thus, the information derived from synergy interactions falls into the task-irrelevant category within each unimodality. We characterize the information that emerges from the combination of two task-irrelevant features as the learned synergy. This distinction is further clarified in Section 3.4.

Question 1b. How to control the learning to make one vector task-relevant and the other task-irrelevant?

Thank you for this inquiry. Our approach draws inspiration from the Variational Information Bottleneck (VIB) technique [2], where we utilize unimodal task-relevant features to finish specific tasks. To effectively separate task-relevant and task-irrelevant information, we incorporate an additional task-related loss function. As detailed in Equation 13, the VIB framework allows us to reduce the information flow between representation $Z^{(m)}$ into two decoupled components, task-related vector $T^{(m)}$ and task-irrelated vector $V^{(m)}$. By minimizing the mutual information terms, $I(Z^{(m)}; T^{(m)})$ and $I(Z^{(m)}; V^{(m)})$, we can reduce the consistent information between $T^{(m)}$ and $V^{(m)}$. Hence, the two vectors can be decoupled and represent task-relevant and task-irrelevant information, respectively.

Question 1c. How to warm-up the encoder.

Thank you for your thoughtful comment. In our method, we specifically warm-up the unimodal encoder, which is crucial for the process of transforming $X^{(m)}$ into $Z^{(m)}$. It is important to note that the variational encoder does not require warming up, a distinction we have clarified in the revised manuscript. The rationale behind warming up the unimodal encoder stems from the observation that, in the early stages of learning, these encoders are not yet capable of extracting specific information effectively. Initiating the learning process without a warm-up phase often results in suboptimal performance due to premature decomposition. To address this, we train every modality respectively with a few epochs to the target, similar to the modality ensemble paradigm. This approach ensures that each modality’s encoder is adequately prepared to extract and subsequently decompose the information effectively.

Question 1d. Explanation of stage 2.

Thank you for your question. In this stage, our objective is to refine the decomposition process to effectively distinguish between different types of interactions. To achieve this, we freeze the unimodal encoder, $\phi^{(m)}$, which allows us to focus solely on training the decomposition network without the interference of evolving unimodal representations. We have updated and clarified this process in Section 3.4.

Question 1e. About reconstruction objective.

Thank you for your question. Indeed, we incorporate a reconstruction loss as part of our learning objective, in line with other VAE-based decomposition methods. This reconstruction loss is essential for minimizing the information loss that occurs after processing by the variational encoder. For a detailed explanation of our model, please refer to Appendix B.2 in the revised manuscript.

Question 1f. How to fine-tune.

This is an excellent point. In this paper, we employ a straightforward yet effective method for fine-tuning: we directly integrate the interaction variables and use them to complete the specified task. Specifically, we concatenate these variables and project them into a unified space, which is then used to complete the task. This approach allows every interaction in the data to be effectively represented and utilized.

Dear Reviewer a4Ud,

Thanks again for your insightful suggestions and comments. We would appreciate knowing if our responses have fully addressed your concerns. We are happy to answer any further questions or comments you may have.

Dear Reviewer a4Ud,

We would like to sincerely thank you for your time and effort in reviewing our paper and providing invaluable feedback. In response to your suggestions, we have clarified the method architecture and provided a more detailed explanation of the experimental setup, along with extended experimental results in the revised manuscript.

If you have any further questions or require additional clarification, we would greatly appreciate it if you could inform us before the rebuttal period ends (less than one day remaining).

Thank you once again for your insightful comments.

Best regards,

The Authors

Question 3. Ablation study setting.

Thank you for your comment. The primary goal of our ablation study is to verify the effectiveness of each module within our proposed decomposition method. Specifically, we aim to understand how these modules perform across various datasets and various data scales. We conducted extensive experiments on the CMU-MOSEI dataset, which includes both audio and text (A+T) and Visual+Text (V+T) modalities. The results in the Table below also demonstrate the effectiveness of each module. We have updated the results in Appendix B.3.

Question 4. Extended to modality larger than 2.

Thank you for this valuable suggestion. Our proposed Decomposition-based Multimodal Interaction learning (DMI) approach is adaptable to scenarios involving three modalities. By implementing only the Task-related Decomposition on DMI (DMI-TD, illustrated in Figure 4), we can extend our framework to accommodate three modalities. We have conducted empirical evaluations on two datasets that each incorporate three modalities: MOSEI, which includes Visual (V), Audio (A), and Text (T) modalities, and UCF101, which consists of RGB, Optical Flow (OF), and Frame Difference (Diff) modalities. Detailed results of these experiments are presented in Appendix B.3.3, verifying that our method remains effective when extending to three modalities.

[1] K. Soomro, “Ucf101: A dataset of 101 human actions classes from videos in the wild,” arXiv preprint arXiv:1212.0402, 2012.

[2] M. K. Hasan, W. Rahman, A. Zadeh, J. Zhong, M. I. Tanveer, L.-P. Morency et al., “Ur-funny: A multimodal language dataset for understanding humor,” arXiv preprint arXiv:1904.06618, 2019.

[3] P. L. De Jager, Y. Ma, C. McCabe, J. Xu, B. N. Vardarajan, D. Felsky, H.-U. Klein, C. C. White, M. A. Peters, B. Lodgson et al., “A multi-omic atlas of the human frontal cortex for aging and alzheimer’s disease research,” Scientific data, vol. 5, no. 1, pp. 1–13, 2018.

[4] H. Chen, W. Xie, A. Vedaldi, and A. Zisserman, “Vggsound: A large-scale audio-visual dataset,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 721–725.

[5] Y. Fan, W. Xu, H. Wang, J. Wang, and S. Guo, “Pmr: Prototypical modal rebalance for multimodal learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 20 029–20 038.

[6] P. P. Liang, Y. Lyu, X. Fan, Z. Wu, Y. Cheng, J. Wu, L. Chen, P. Wu, M. A. Lee, Y. Zhu et al., “Multibench: Multiscale benchmarks for multimodal representation learning,” arXiv preprint arXiv:2107.07502, 2021.

[7] P. Xu, X. Zhu, and D. A. Clifton, “Multimodal learning with transformers: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.

Question 1. Lack for Holistic Experimental Results

Thank you for your valuable suggestions regarding the holistic validation of our methods. In response, we have incorporated extensive experiments in the revised manuscript, aligning with your recommendations. Below, we summarize the key questions addressed in these experiments:

Question 1a. More modalities.

In addition to our initial audio-visual methods, we expand our experiments to include additional modalities. Specifically, we validate RGB + Optical Flow on the UCF101 dataset [1] for action recognition, Audio + Text on the UR-FUNNY dataset [2] for humor detection, and mRNA + methylation data on the ROSMAP dataset [3] for Alzheimer’s Disease diagnosis. We updated Table 2 to replace the synthetic dataset AV-MNIST with the UCF-101 dataset to better showcase the effectiveness of our method across real-world datasets, and add other experiments in Appendix B.3. These experimental results demonstrate the flexibility of our approach across different modalities and complex tasks.

Question 1b. Richer temporal information.

This is an insightful question of considering richer temporal information. To address this, we expanded our application of the method across more frames within the CRMEA-D and KS datasets. Our findings indicate that increased richer temporal information enhances task performance to some degree. Moreover, our propose DMI paradigm still demonstrates improvements with this abdundant temporal information. Detailed results and analyses are provided in Appendix B.3 of the revised manuscript.

Question 1c. Larger scale.

After careful consideration of time and feasibility, we choose the VGGsound dataset[4], which encompasses over 210,000 entries across 309 categories. Detailed results are presented in the following table and further detailed in Appendix B. These results demonstrate that our DMI method significantly outperforms existing approaches on large-scale datasets.

Question 1d. Different backbone.

The chosen backbones, ResNet and LeNet, are widely utilized in multimodal research and are consistent with prior studies [5,6]. Also, we validate our method on the Hierarchical Multimodal Transformer backbone [7] as detailed in Table 3, which serves to further validate the scope of our approach. Additionally, we conducted expanded experiments on the KS dataset using ResNet34 and on the CMU-MOSEI dataset using an LSTM backbone. These experiments further verify the effectiveness of our method. Detailed results are presented in Appendix B.3.

Question 2. Specific differences in MMML.

Thank you for your comment. To ensure a fair comparison across different methods, we standardized the backbone across all approaches, followed by [6], applying distinct strategies of different methods to this common framework. For the MMML approach, we incorporated its fusion module on the aligned backbone. This module includes both an attention mechanism and a multi-loss strategy.

Dear oH8U,

Thank you for your thoughtful review and constructive feedback. We have carefully reviewed your comments and made the necessary adjustments. We would appreciate it if you could confirm whether the revisions address your concerns. We would be happy to provide additional information if any further clarification is required.

Dear Reviewer oH8U,

We would like to sincerely thank you for reviewing our paper and providing valuable feedback. In response to your suggestion, we have added experimental results and clarified the experimental setting in the revised manuscript.

Please feel free to reach out if you have any further questions or require additional clarification before the rebuttal period concludes (less than one day remains).

Thank you once again for your insightful comments.

Best regards,

The Authors

Question 4. Other paradigms over joint learning and modality ensemble.

Thank you for highlighting these points. In supervised learning that involves multiple modalities, the central challenge lies in leveraging multimodal data effectively to accomplish multimodal tasks. According to previous literature [1], joint learning of multiple modalities in a shared space and separate learning within each modality's own space are two primary paradigms in multimodal learning. Based on this, two prevalent paradigms—joint learning and modality ensemble—represent different approaches to this challenge. The modality ensemble approach utilizes individual modalities independently to complete tasks, whereas joint learning integrates all modalities collectively for task completion. Additionally, in the domain of self-supervised learning, alignment, and contrastive learning are commonly employed paradigms. The insights gained from analyses within these paradigms are invaluable and will be considered in our future research.

[1] T. Baltrusaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE Transactions on pattern analysis and machine intelligence, vol. 41, no. 2, pp. 423–443, 2018.

Question 1. Expression in formulation

Question 1a. Theoretical contributions of Lemma 3.3.

Thank you for your insightful comment. Lemma 3.3 describes that under redundancy-type interactions, the modality ensemble approach can reduce the generalization gap between the theoretical mutual information, $I(Z;Y|**c**)$, and the empirical mutual information, $I_S(Z;Y|**c**)$. The ensemble paradigm aims to solely learn from each unimodality to complete the task. Consequently, we introduce $\tilde{S}$ (referenced in Lemma 3.3) to provide a more accurate representation of the learning paradigm. Due to the nature of redundancy interaction, both modality contains sufficient task-related information. Therefore, every sample within $\tilde{S}$ contributes to training the multimodal task.  With $2n$ samples available in $\tilde{S}$, according to Proposition 3.2, a reduction in the generalization gap is achieved. This reduction in the generalization gap implies that enhancing the empirical mutual information contributes to an increase in the theoretical mutual information, similar to the Probably Approximately Correct (PAC) learning theory. Hence, we present in Table 1 that modality ensemble achieves a tighter upper bound than joint learning under redundancy-type interactions.

Question 1b. Confusion of big-O notion

Thank you for your invaluable comment. Our analysis of Big-O notation is to illustrate the differences in the upper bounds of the generalization gap (as defined in Equation 8) between the modality ensemble and joint learning paradigms. This presentation may have lacked clarity. To address this, we have revised the relevant equation to improve both its clarity and simplicity. The revised of Table 1 is presented below:

The revised table now more accurately represents the differences, emphasizing that the main variance in redundancy is the factor $\sqrt{\frac{\omega}{n}}$ for joint learning compared to $\sqrt{\frac{\omega}{2n}}$ for modality ensemble. Further details of the modifications are discussed on Page 5, Section 3.3 of our manuscript. Thank you again for this invaluable comment.

Question 2. Explanation of model architecture.

Thank you for raising this point. The architecture of our proposed DMI (see Figure 3) consists of unimodal encoder $\phi^{(m)}$ to obtain unimodal representation $Z^{(m)}$, and two decomposition modules, decomposing the representation into different interactions $R, U^{(1)}, U^{(2)}, S$. The unimodal encoder varies for different tasks. The decomposition module is architecture like Variational Autoencoder (VAE). Each decomposition module is structured on a VAE framework, where the encoders, composed of Multi-Layer Perceptrons (MLPs), predict the mean and variance. Conversely, the decoders are designed as multilayer networks to ensure minimal information loss during the decomposition process. The alignment of features across modalities is enforced by minimizing the Kullback-Leibler (KL) divergence between the corresponding distributions. We add more detailed descriptions in Appendix B.2.

Question 3. The idea paper conveys.

Thank you for your insightful observation. In addition to your mention of how interaction decomposition improves multimodal interaction learning, we have provided a comprehensive theoretical analysis that underscores the importance of considering various multimodal interactions within multimodal learning. This analysis explains the crucial role that learning from holistic interactions helps better multimodal learning performance. It also illuminates the underlying mechanics of our interaction decomposition method, further justifying its application.

Dear Reviewer HrvF,

Thank you once again for your valuable comments and suggestions. We would appreciate your confirmation on whether our responses have addressed your concerns. Please let us know if you have any further questions or comments.

Dear Reviewer HrvF,

We would like to sincerely thank you for reviewing our paper and providing invaluable feedback. Your insights have greatly contributed to enhancing the quality of the manuscript.

If there are any further points that require clarification, we would be grateful if you could let us know before the end of the rebuttal period (less than 1 day remaining).

Thank you once again for your thoughtful and constructive comments.

Best regards,

The Authors