Neuro-Inspired Information-Theoretic Hierarchical Perception for Multimodal Learning

Thank you for acknowledging the novelty and sensibility of our Information-Theoretic Hierarchical Perception (ITHP) model, as well as for recognizing the strength of our experimental results. We also greatly value your insightful comments, which have guided us in further enhancing our paper. Here we provide the point-wise response to your concerns:

Algorithms for defining the order of modalities: Thank you for pointing out your concern on defining the order of the modalities when there are > 3 modalities and we do not have prior information about the multimodal task. The sequence of modalities indeed influences the performance of our model. When the number of modalities increases and there is a lack of relevant prior knowledge, determining the richness of information in each modality and the sequence among them becomes crucial. To address this problem, we have developed practical algorithms to define the order of modalities: 1. We employ Sample Entropy (SampEn) to assess the information richness within different modalities, which serves as the basis for ranking these modalities. 2. For specific downstream tasks, we implement a greedy algorithm in conjunction with a submodular function to more comprehensively determine the order of the modalities. Detailed explanations and algorithms for these methods are provided in the appendix J (page 26-27), as also mentioned in the 'Response to all reviewers' comments above.

Inference latency: Thank you for your valuable suggestion to test the inference latency, which has enabled a more thorough assessment of our model's performance. In response to your concern about the inference latency of our proposed hierarchical architecture as compared to standard non-hierarchical approaches, we conducted a comparative analysis using a simple MLP head. This involved concatenating modalities M0, M1, and M2 and processing them through an MLP head, representative of the predictor component in our ITHP model. The inference latency for this simple configuration was approximately 0.00221 ms per sample. In contrast, our full ITHP model, incorporating a hierarchical structure, recorded an inference latency of 0.0120 ms per sample using the same experiment setting. It's noteworthy that during inference, the detectors in our model are not active, contributing to its overall efficiency. Despite the anticipated increase in latency compared to an MLP head due to its hierarchical nature, our model remains competitive in terms of efficiency. We appreciate your suggestion and have included the above-mentioned discussion in Appendix G (page 24).

Thank you for your thorough review of our paper, we greatly value your insightful comments. Your suggestions have helped us enhance the content of our paper. Here we provide our responses to the concerns you raised.

Regarding to the inspiration from neuroscience research: Our model is inspired by neuroscience research, highlighting how the brain hierarchically processes multimodal information. We have integrated an information bottleneck framework into our design to emulate the hierarchical process, which stands as an innovation in our work and sets our model apart from traditional approaches. Rather than initiating with a fusion structure as is common in conventional models, our model adopts a comprehensive, information theoretic approach.

Sequential processing and order of modalities: We are grateful for your recommendations regarding offering theoretical analysis or practical guidance for determining the order of modalities. Based on your advice and comments from other reviewers, we believe it is necessary to provide practical guidance for defining the order of modalities. We have developed practical algorithms to define the primary modality and the order of modalities: (i) We employ Sample Entropy (SampEn) to assess the information richness within different modalities, which serves as the basis for ranking these modalities. (ii) For specific downstream tasks, we implement a greedy algorithm in conjunction with a submodular function to more comprehensively determine the order of the modalities. Detailed explanations and algorithms for these methods are provided in the appendix J (page 26-27), as also mentioned in the 'Response to all reviewers' comments above.

Based on the algorithms mentioned above, we validated the modal selection order used in our sarcasm detection experiments. The computed sample entropies for the V/T/A modalities are 2.388/2.063/0.071, respectively. This indicates that the ranking of information richness for these modalities should be V→T→A. The outcome derived from applying the submodular function method also indicates a V→T→A sequence. The results of corresponding experiments for varying primary modalities are presented in Section 5 (page 9) and the results are shown in Appendix F.5 (Page 23). We also present the results below. This table illustrates the effects of changing the primary modality (modal order) on weighted precision, recall, and F-score.

Table 1. Varying Orders of Modalities.

Table 1 showcases the impact of changing the order of modalities on the weighted Precision, Recall, and F-Score across both sarcastic and non-sarcastic classes, averaged across five folds. Underlined values highlight the best results for each metric. The modalities are denoted as: $T$ for text, $A$ for audio, and $V$ for video. The results show that the order of modalities, particularly the designation of the primary one, considerably influences the performance of the model. The results also indicate that the order of modalities should be V→T→A, which is consistent with the conclusions drawn from our algorithm.

Thank you for your prompt response and for acknowledging the resolution of the order of modalities concern. Here we provide the point-wise response to your remaining concerns:

1. Regarding your concern about the sequential pipeline versus fusion of all modalities once together, we wish to highlight that, while other state-of-the-art baseline models adopt strategies to fuse all modalities at once, the experiments have demonstrated the superiority of our hierarchical approach (refer to Table 1,2 and 3 in the main body of the article, Table 4 in Appendix E, Table 5 in Appendix F.1 and Table 6 in Appendix F.3).

2. Your point about the potential loss of complementary information in the third modality is well-taken. Our method, however, ensures that no modality is neglected. Specifically, the information bottleneck is applied not only between the state $B_0$ and the second modality $X_1$ but also between the state $B_1$ and the third modality $X_2$. The entire process is data-driven, and the final information flow considers all modalities, representing the optimal solution. This guarantees that all modalities are considered, utilizing the information of the third modality (please refer to Appendix C (Pages 14-17) for the detailed design of our model).

3. Our sequential pipeline design is not arbitrary but is grounded in neuroscience principles. We believe that the hierarchical design allows for a more natural and robust integration of modalities, which is evidenced in our experiment results compared with the baseline fusion models. We also provide the detailed explanation and theoretical basis for designing our hierarchical model based on the information bottleneck in Appendix C (Pages 14-17).

We hope this response has addressed your concerns, and we sincerely appreciate your feedback, which is instrumental in enhancing the clarity of our work. We genuinely value your contributions and assure you that any further concerns will be carefully considered.

Thank you for a thorough review of our paper and for recognizing the fundamental premise of our research. We also appreciate your recommendations of related works [1,2,3]. In our revised manuscript, we have expanded the discussion on related literature to appeal to a wider audience. Below, we provide the detailed point-wise response to address your concerns.

Comparative analysis of additional methods for sarcasm detection: Thank you for sharing the related works [2,3] on sarcasm detection. Specifically, HFM [2] proposed a task-oriented hierarchical fusion model, which incorporates early fusion, representation fusion, and modality fusion techniques to enhance the comprehension and representation of each modality. We tested HFM using the MUStARD dataset, applying its modality fusion module under the same experimental conditions as our original study. The results, now included in Table 6 of Appendix F.3 (page 21) of the revised manuscript, are as follows:

Table 1: Results of Sarcasm Detection on the MUStARD dataset.

This table displays the weighted Precision, Recall, and F-Score for the V-T-A (video, text, audio) modality, averaged across five folds. Bold figures represent the highest scores achieved. As the results indicate, our ITHP model demonstrates a notable increase of 8.5% in precision, 9.3% in recall, and 9.1% in F-score compared to the HFM [2] model. These improvements underscore the superiority of our model in sarcasm detection.

The DIP [3] model processes inputs consisting of two modalities: images and text. It computes a relation matrix of their feature vectors, which is then fed into two modules, SSC and SID, for further processing and eventual concat for prediction. This model architecture, specifically its requirement for modality correlation matrix computation and the use of two distinct branches for sentiment analysis, does not align well with our sarcasm datasets. Consequently, it is not feasible to conduct equivalent comparative experiments. Regardless, we appreciate your sharing of the related works, and we have also discussed them in our related work in Appendix B (page 14) to reach a broader audience as follows:

“The latest work in the field of sarcasm detection has expanded our research horizons. The HFM [2] utilizes early fusion and representation fusion techniques to refine feature representations for each modality. It integrates information from various modalities to better utilize the available data and enhance sarcasm detection performance. The DIP [3] creatively adopts dual incongruity perception at factual and affective levels, enabling effective detection of sarcasm in multi-modal data.”

Literature survey update: Thank you for introducing us to the exceptional work [1], it will greatly enhance our literature review. Diverging from the prevalent static fusion approaches, DynMM presents an innovative perspective by adaptively fusing multimodal data and creating data-dependent forward paths during inference. The main highlight of the DynMM model lies in its ability to significantly reduce computational resource consumption while maintaining relatively high model performance.

We also compared the performance of our ITHP model with DynMM on the CMU-MOSEI dataset. DynMM achieved an accuracy of 79.8% and a Mean Absolute Error (MAE) of 0.60. In contrast, our model achieved an accuracy of 87.3% and an MAE of 0.56, surpassing DynMM by over 9.4% in accuracy and exhibiting a 6.7% lower MAE. These results further underscore the superiority of our model in terms of accuracy. We have also discuss DynMM in in Appendix B (page 14):

“The DynMM [1] creates the possibility of reducing computing resource consumption. By introducing dynamic multi-modal fusion, which fuses input data from multiple modalities adaptively, leading to reduced computation, improved representation power, and enhanced robustness.”

[1] Xue Z, Marculescu R. Dynamic multimodal fusion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2023 (pp. 2574-2583).

[2] Cai Y, Cai H, Wan X. Multi-modal sarcasm detection in twitter with hierarchical fusion model. InProceedings of the 57th annual meeting of the association for computational linguistics 2019 Jul (pp. 2506-2515).

[3] Wen C, Jia G, Yang J. DIP: Dual Incongruity Perceiving Network for Sarcasm Detection. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2023 (pp. 2540-2550).

We sincerely thank you for highlighting the good motivations behind our models and recognizing how their integration of neuroscience principles adds a novel perspective to multimodal learning. We also thank you for sharing these two interesting related works [1,2], which we have discussed in the revised version to reach a broader audience. Additionally, we provide detailed answers to address your questions below.

Regarding the comparison between our method, ITHP, against SOTA methods such as UniMSE and SPECTRA: Thank you for sharing these related works [1,2], we have first added them into the references and discussed them in the related work section in Appendix B (page 14) as follows: “The current state-of-the-art model presents novel insights, UniMSE [1] innovatively integrates multimodal sentiment analysis and emotion recognition in conversations into a unified framework, enhancing prediction accuracy by leveraging the synergies and complementary aspects of sentiments and emotions. SPECTRA [2] is pioneering in spoken dialog understanding by pre-training on speech and text, featuring a unique temporal position prediction task for precise speech-text alignment .”

We have also expanded our baselines to include more recent works such as UniMSE [1], MIB [3] and BBFN [4] for a more comprehensive comparison, as detailed in the updated Table 2 (page 8) in the paper. The results show that our model continues to surpass the baselines.

Regarding the comparison with the models you mentioned, particularly UniMSE [1] and SPECTRA [2]: First, regarding performance, we outperform UniMSE on CMU-MOSEI and CMU-MOSI, two common evaluation benchmarks shared by both of the works, by around 1% and 3% points in terms of F1 score. Meanwhile, SPECTRA is surpassed by 1% in MOSI datasets, demonstrating our model’s superiority. Here we would like to point out that our data preprocessing is based on the previous paper MAG [5], which only provides off-the-shelf representations of the images and audio. However, both UniMSE and SPECTRA, adopted a different pipeline (e.g., in UniMSE, they use more powerful tools such as effecientNet to extract features from the video, coupled with additional A/V-sLSTM to further distill the multimodal signals). In this sense, the comparison with Self-MM, MMIM, and MAG are considered as more fair baselines given the same pre-processing pipelines, where ITHP has dominated in performance against all of the above. Second, our ITHP is an information-theoretic with guaranteed proof of aggregating the most useful information across different modalities. Previous works, such as UniMSE and SPECTRA, either empirically instead of theoretically adopts contrastive loss to force the latent space to be as distinctive as possible, or empirically aligns sequential audio-text features which are not generalizable to images.

[1] UniMSE : G. Hu, et al., UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition, EMNLP 2022

[2] SPECTRA : T. Yu, et al., Speech-Text Dialog Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment, ACL 2023

[3] MIB: Mai S, Zeng Y, Hu H. Multimodal information bottleneck: Learning minimal sufficient unimodal and multimodal representations. IEEE Transactions on Multimedia. 2022.

[4] Han, Wei, et al. "Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis." Proceedings of the 2021 International Conference on Multimodal Interaction. 2021.

[5] Rahman, Wasifur, et al. "Integrating multimodal information in large pretrained transformers." Proceedings of the conference. Association for Computational Linguistics. Meeting. Vol. 2020. NIH Public Access, 2020.

Thank you for your meticulous review and valuable insights. We sincerely appreciate your recognition of the ITHP model’s intuitiveness and its strong grounding in information theory. In the following, we offer our detailed responses and clarifications to address the concerns and queries you have highlighted.

The usage of "fusion" in our method: Thank you for highlighting the usage of the term “fusion” in our study. We are grateful for your detailed attention to this aspect, and we have thoroughly reviewed our manuscript to ensure that this term is used accurately and appropriately. Fusion models primarily aim to integrate information from various modalities into a unified representation [1,2]. In terms of this objective, our work can certainly be viewed as a fusion model constructing a compact representation by utilizing information from various modalities. In terms of technique, we indeed adopt a distinct hierarchical approach compared to traditional fusion models, which stands as a significant innovation in our work. Although the secondary modalities serve as detectors in our model, it is worthy noting that, within the deep learning architecture, our model has certain capability to indirectly learn the complementary information from secondary modalities. As explained in Appendix H (page 24):” by harnessing the capabilities of our deep learning framework, our model has a certain ability to learn the complementary information from secondary modalities. To achieve a harmonious representation, we’ve incorporated specific terms in the loss function. Specifically, taking the 3-modalities learning problem as an example, term $L_{1} =−\beta\cdot I\left (B_{0};X_{1} \right)$ is tasked with supervising the knowledge transfer from $X_{1}$ to $B_{0}$, while term $L_{2} =−\gamma\cdot I\left (B_{1};X_{2} \right)$ is responsible for supervising the knowledge dissemination from $X_{2}$ to $B_{1}$. Through this framework, the ITHP model is primed to discern and incorporate complementary information from non-input modalities.” In summary, we prefer to use the term 'fusion' in the context of our mode, and we will make it clear in our revised version.

[1] Gao, Jing, et al. "A survey on deep learning for multimodal data fusion." Neural Computation 32.5 (2020): 829-864.

[2] Atrey, Pradeep K., et al. "Multimodal fusion for multimedia analysis: a survey." Multimedia systems 16 (2010): 345-379.

Experiments with different primary modalities: We appreciate your question regarding the selection of the primary modality and its impact on the ITHP model's performance. In fact, we have already conducted the experiments for varying primary modalities as highlighted in Section 5 (page 9) and the results are shown in Appendix F.5 (Page 23). We also present the results below. This table illustrates the effects of changing the primary modality (modal order) on weighted precision, recall, and F-score.

Table 3. Varying Orders of Modalities.

The table showcases the impact of changing the order of modalities on the weighted Precision, Recall, and F-Score across both sarcastic and non-sarcastic classes, averaged across five folds. Underlined values highlight the best results for each metric. The modalities are denoted as: $T$ for text, $A$ for audio, and $V$ for video.

The results show that the order of modalities, particularly the designation of the primary one, considerably influences the performance of the model. In many cases, we can rely on prior knowledge to guide the selection of the primary modality and its sequence. However, when such knowledge is lacking, effective methods are essential for determining these aspects. In response to the feedback from reviewers, we have developed practical algorithms to define the primary modality and the order of modalities: (i) We employ Adapting Sample Entropy (SampEn) to assess the information richness within different modalities, which serves as the basis for ranking these modalities. (ii) For specific downstream tasks, we implement a greedy algorithm in conjunction with a submodular function to more comprehensively determine the order of the modalities. Detailed explanations and algorithms for these methods are provided in the appendix J (page 26-27), as also mentioned in the 'Response to all reviewers' comments above.

Based on the methods for modalities ranking we proposed, we validated the modal selection order used in our experiments. For the first method, the computed SampEn for the V/T/A modalities are 2.388/2.063/0.071, respectively. This indicates that the ranking of information richness for these modalities should be V→T→A. Additionally, by applying the algorithm (ii), we obtained the same modality order of V→T→A. Hence, we chose V as our primary modality. This aligns with the results obtained in our experiments, confirming the viability of our proposed detection criteria.

Learning complementary information: As we mentioned in the above response (The usage of "fusion" in our method), our ITHP model, built within the deep learning architecture has a certain ability to learn the complementary information from secondary modalities. Please refer to Appendix H (page 24) for details.

Model performance in the presence of noise: Thanks for the insightful suggestions on adding noise to the modality $M_{p}$ and modality $M_{1}$. We believe these experiments will bring valuable insights to further elucidate our model. Theoretically, our model filters redundant information within its information-bottleneck hierarchical structure, and we anticipate it will have advantages over traditional fusion models in terms of reducing the impact of noise. According to your suggestions, we conducted the noise addition experiments as follows:

In the sarcasm detection dataset, we introduced noise to the normalized data of both the primary and secondary modalities separately to test the model's resistance to noise interference. We apply a series of Gaussian noises with a mean of 0 and a varying standard deviation ranging from 0.05 to 0.3, increasing in increments of 0.05. For each noise level, we repeated the experiment 50 times and calculated the average result. We have included these experiments in the revised version (Please refer to Appendix F.6, page 23-24), and the results are presented below:

Table 1. Evaluating Gaussian Noise Impact on Sarcasm Detection in Text Modality.

Table 1 presents the weighted precision and recall for sarcasm detection on the MUStARD dataset, with an analysis of Gaussian noise addition to the text ($M_{1}$) modality. Results are averaged over five cross-validation folds and 50 trials for each level of noise intensity. Each column indicates the standard deviation (represented by $\sigma$) of the introduced Gaussian noise with a mean of 0. Changes in performance metrics relative to the noise-free baseline are expressed as percentages

Table 2. Evaluating Gaussian Noise Impact on Sarcasm Detection in Video Modality.

This table presents the weighted precision and recall for sarcasm detection on the MUStARD dataset, with an analysis of Gaussian noise addition to the video ($M_{p}$) modality. Results are averaged over five cross-validation folds and 50 trials for each level of noise intensity. Each column indicates the standard deviation (represented by $\sigma$) of the introduced Gaussian noise with a mean of 0. Changes in performance metrics relative to the noise-free baseline are expressed as percentages

The results shown in Table 1 and Table 2 demonstrate that, compared to the baseline model, our model exhibits greater robustness to noise interference. The impact of adding noise to the primary modality is more significant than the effect of noise on the secondary modality.

(ii) Ranking modalities using submodular functions

Given a set of $N$ different modalities $M = \{m_1, m_2, \ldots, m_N\}$, the goal is to find a subset of modalities $S \subseteq M$ that maximizes the predictive performance in a given downstream task. Submodular Function Definition: Let $f: 2^M \rightarrow \mathbb{R}$ be a submodular function that maps a set of modalities to a real number, indicating their performance in a predictive task. For any $A \subseteq B \subseteq M$ and $x \notin B$, the function $f$ satisfies:

$f(A \cup \{x\}) - f(A) \geq f(B \cup \{x\}) - f(B)$

Greedy Algorithm: The greedy algorithm starts with an empty set, and iteratively adds a modality that maximizes the function $f(S)$:

$S_{i+1} = S_i \cup \{\arg\max_{x \notin S_i} f(S_i \cup \{x\})\}$

---

\ $ & \**Algorithm 2: Greedy Algorithm for Ranking Modalities** \\\\ & \**function** \\ \\text{GreedySubmodularSort}(modalities, evaluatePerformance) \\\\ & S \\gets \\text{empty set} \\\\ & ModalityList \\gets \\text{empty list} \\\\ & \**while** \\ |S| < | modalities | \\ \**do** \\\\ & \\quad bestImprovement \\gets -\\infty \\\\ & \\quad bestModality \\gets \\text{null} \\\\ & \\quad \**for each** \\ modality \\ \\text{ in modalities} - S \\ \**do** \\\\ & \\qquad newPerformance \\gets evaluatePerformance(S \\cup \\{modality\\}) \\\\ & \\qquad improvement \\gets newPerformance - evaluatePerformance(S) \\\\ & \\qquad \**if** \\ improvement > bestImprovement \\ \**then** \\\\ & \\qquad \\quad bestImprovement \\gets improvement \\\\ & \\qquad \\quad bestModality \\gets modality \\\\ & \\qquad \**end if** \\\\ & \\quad \**end for** \\\\ & \\quad \**if** \\ bestModality \\ \\text{is null} \\ \**then** \\\\ & \\quad \\quad \**break** \\\\ & \\quad \**end if** \\\\ & \\quad S \\gets S \\cup \\{bestModality\\} \\\\ & \\quad ModalityList.\\text{append}(bestModality) \\\\ & \**end while** \\\\ & \**return** \\ ModalityList \\\\ & \**end function** \\\\ \ $

---

[1] Cai Y, Cai H, Wan X. Multi-modal sarcasm detection in twitter with hierarchical fusion model. InProceedings of the 57th annual meeting of the association for computational linguistics 2019 Jul (pp. 2506-2515).

[2] G. Hu, et al., UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition, EMNLP 2022

[3] Mai S, Zeng Y, Hu H. Multimodal information bottleneck: Learning minimal sufficient unimodal and multimodal representations. IEEE Transactions on Multimedia. 2022.

[4] Han, Wei, et al. "Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis." Proceedings of the 2021 International Conference on Multimodal Interaction. 2021.

[5] Zurek, Sebastian, et al. "On the relation between correlation dimension, approximate entropy and sample entropy parameters, and a fast algorithm for their calculation." Physica A: Statistical Mechanics and its Applications 391.24 (2012): 6601-6610.

[6] Pincus, Steven M. "Approximate entropy as a measure of system complexity." Proceedings of the National Academy of Sciences 88.6 (1991): 2297-2301.

[7] Richman, Joshua S., and J. Randall Moorman. "Physiological time-series analysis using approximate entropy and sample entropy." American journal of physiology-heart and circulatory physiology 278.6 (2000): H2039-H2049.

[8] Krause, Andreas, and Daniel Golovin. "Submodular function maximization." Tractability 3.71-104 (2014): 3.