Hyper-Modality Enhancement for Multimodal Sentiment Analysis with Missing Modalities

We sincerely thank Reviewer bLRy for your time and insightful feedback. We are especially grateful for your recognition of the thoughtful design of our proposed framework. Your comments have significantly helped us deepen our analysis and improve the clarity of our paper. Below, we address each of your concerns in detail.

Weaknesses & Questions:

W1: Batch Size Sensitivity & Qualitative Examples of Enhancement

(1) Thank you for highlighting the potential impact of batch size on our HME. We agree that this is a crucial aspect that can offer valuable insights into HME’s robustness and practical deployment. To explore this, we conducted experiments on the MOSI dataset under the random missing protocol, evaluating batch sizes of 64, 128, and 256. The results are summarized in Tables 1 and 2 below (to be included in the appendix). We observed that larger batch sizes consistently lead to better performance. This is likely because a larger batch offers more diverse and semantically similar samples, allowing the enhancement module to extract richer information. These findings will be included in the revised appendix.

Table 1: Accuracy performance on MOSI with random missing protocol

Table 2: F1 performance on MOSI with random missing protocol

(2) As for qualitative examples of enhancement, We appreciate your suggestion to include more qualitative examples. While Figure 6 in appendix currently shows some case visualizations from the MOSI dataset, we agree that it would be beneficial to expand this analysis. In the revised version, we will include visualizations under varying batch sizes and similarity thresholds, highlighting which samples are selected for enhancement and why. This will provide readers with a more concrete grasp of how and why enhancement contributes to performance.

W2: Interpretation of Learnable Prompts

Thank you for your insightful comment regarding the prompts used in the Perceiver and hyper-modality interaction modules. Inspired by your suggestion, we realized that simple visualization (as shown in Figures 3b and 5) is insufficient to fully convey their function. We will enhance our qualitative analysis by including:

• Visualization of how changing the similarity threshold affects prompt behavior and sample selection,
• Analysis of attention weights assigned to each prompt during enhancement,
• Measurements of distances between original, enhanced, and hyper-modality embeddings.

These analyses will clarify the function and utility of prompts in attending to sentiment-relevant information and will help distinguish the role of hyper-modality from raw or unimodal representations.

W3: Detailed Ablation on the Variational Information Bottleneck (VIB)

We thank you for encouraging a deeper investigation into the role of the VIB module. Following your suggestion, we conducted a more granular analysis of $I(Z,X)$ and $I(Z,Y)$. Specifically, we varied their weights individually and jointly. As shown in Table 3 below, increasing the weight of $I(Z,X)$ compresses representations too aggressively, degrading performance by removing sentiment-relevant features. Conversely, increasing $I(Z,Y)$ improves performance up to a point, after which it declines—likely due to overfitting or underemphasizing alignment with input features. Jointly increasing both follows a similar trend.

Table 3: Performance comparison under different weights of VIB components

Additionally, to further demonstrate the effectiveness of VIB, we provide an ablation study comparing HME with and without the VIB component across various missing rates and datasets, as shown in Tables 6 and 7 in the appendix. These results consistently show performance drops when VIB is removed, underscoring its contribution to robustness and representation quality.

Q1: Experimental Setting Clarification

Thank you for pointing this out. We clarify that in the fixed missing modality setting (e.g., when “L” is missing), the modality is only removed during validation and inference—not during training. This setup aligns with prior works (GCNet (TPAMI 2023), MPLMM (ACL 2024), DiCMoR (ICCV 2023), IMDer (NeurIPS 2023), and LNLN (NeurIPS 2024)) and is intended to simulate real-world scenarios where certain modalities may be missing only at test time. In contrast, under the random missing protocol, the same missing rate is applied uniformly during training, validation, and testing. We will revise the paper to state this more explicitly.

Q2: Clarification of Figure 3b (Section 4.4)

We apologize for the lack of clarity in our explanation. Figure 3b, when considered alongside Figure 5 (in the appendix), provides a more complete picture. In Figure 3b, the t-SNE plot shows that the hyper-modality representations ($F_m$) form more distinct sentiment-aware clusters than the original unimodal representations ($H_m$), indicating that $F_m$ captures more discriminative information. Figure 5 further demonstrates that after applying uncertainty-aware fusion, the $F_m$ representation is effectively integrated with the original $H_m$, resulting in more structured and separable clusters. Together, these visualizations support our claim that the hyper-modality enhances sentiment-relevant information in a meaningful way.

Once again, we sincerely thank the reviewer for your detailed and thoughtful feedback, which has helped us improve both the depth and clarity of our paper. We will incorporate these enhancements into our revision.

We sincerely thank Reviewer py7h for the time and effort dedicated to reviewing our paper. We deeply appreciate your recognition of our work as "a technically sound and well-motivated framework", and we value the insightful comments and constructive suggestions you provided. Below, we respond to each of your concerns in detail.

Weaknesses & Questions:

W2&Q1: Interaction Between VIB and Uncertainty-Aware Fusion

Thank you for highlighting this important aspect. Your comments helped us further explore the interplay between the variational information bottleneck (VIB) and the uncertainty-aware fusion.

(1) The VIB module is specifically introduced to filter out misleading semantic signals that may appear during cross-sample enhancement. For example, as shown in Table 10 (Appendix), the selected similar samples may contain sentiment-relevant cues that contradict the target sample’s sentiment label, leading to performance degradation. Rather than aggregating all available signals, VIB helps retain only reliable features. The ablation results (Tables 6 and 7 in Appendix) show that removing VIB results in a consistent performance drop across all datasets, indicating its effectiveness in suppressing noise and misleading cues.

(2) Motivated by your question on the trade-off between $I(Z,X)$ and $I(Z,Y)$, we conducted a detailed study by varying their weights individually and jointly. Results (see Table 1 below) show that increasing the $I(Z,X)$ weight (i.e., stronger compression) tends to suppress useful sentiment information, degrading performance. Conversely, while increasing $I(Z,Y)$ initially improves results, excessive emphasis eventually hurts performance—likely due to under-utilization of complementary features or overfitting.

Table 1: Performance comparison under different weights of VIB components

W1&Q2: Evaluation Under Real-World Missingness and Cross-Domain Settings

We agree that assessing performance under natural missingness and domain shifts is crucial for real-world applicability. Due to the lack of standard datasets with naturally missing modalities (e.g., social media posts with missing images or audio), we followed existing work (IMDer (NeurIPS 2023) and LNLN (NeurIPS 2024)) in simulating missingness. However, HME differs fundamentally by not relying on ground-truth representations of the missing modalities, aiming for more realistic simulation.

For domain-shifts scenarios, we have extended our experiments to scenarios using the MUStARD and UR-FUNNY sarcasm detection datasets. Results (Tables 2–5 below) show that HME consistently outperforms baseline across various missing-modality conditions, demonstrating improved robustness in domain-shift settings.

As for cross-dataset generalization, we faced challenges due to label inconsistency (e.g., binary vs. four-class sentiment in MOSI/MOSEI vs. IEMOCAP) and differing input dimensions (e.g., (5/20 vs. 74/35) of audio/video features in MOSI vs. MOSEI). Nonetheless, we acknowledge its importance and will extend our experiments using the MUStARD and UR-FUNNY sarcasm detection datasets. We will include these findings and additional experiments in our revised manuscript.

Table 2: Accuracy performance on MUStARD with fixed missing protocol

Table 3: F1 performance on MUStARD with fixed missing protocol

Table 4: Accuracy performance on UR-FUNNY with fixed missing protocol

Table 5: F1 performance on UR-FUNNY with fixed missing protocol

Q3: Choice of cosine similarity for cross-sample selection

Thank you for this valuable suggestion. We chose cosine similarity due to its robustness to curse of dimensionality and its widespread use in related work. Moreover, many contrastive learning methods—including CLIP—also rely on cosine similarity as a similarity metric.

In our case, cross-sample selection must be unsupervised, since real-world missingness often occurs in settings without label access. Unlike contrastive learning approaches that require carefully curated positive/negative pairs or rely on known modality alignment (as in CLIP), HME is designed to flexibly select multiple relevant samples without supervision, improving adaptability across diverse applications.

Q4: Relationship between Eq. 5 and Bayesian Deep Learning (BDL) uncertainty

Equation 5 draws inspiration from uncertainty quantification principles in BDL, particularly in its use of variance to estimate uncertainty. However, while BDL typically estimates model uncertainty via techniques like Monte Carlo dropout, our method estimates uncertainty at the representation level—specifically, the robustness of VIB-compressed features. Thus, Eq. 5 shares a conceptual link with BDL, but it serves a distinct purpose in modulating the fusion of original and enhanced modalities based on feature reliability.

Q5: Ethical risks of emotion inference from incomplete data

We fully acknowledge the ethical implications of inferring emotions from incomplete inputs. Inaccurate emotion recognition can lead to misrepresentation, particularly of marginalized groups, or raise concerns about profiling and privacy. While our method aims to reduce such risks by improving robustness to missing modalities, we recognize the importance of transparent and responsible deployment. We will explicitly discuss these concerns and potential safeguards in our revised manuscript.

W3: Visualizing hyper-modality representations

Thank you for pointing out the need for clearer qualitative explanations of our hyper-modality representations. While we include relevant visualizations (e.g., Figure 3b, Figure 5, and Figure 6 in the appendix), we agree that these could be more illustrative.

To address this, we plan to extend our qualitative analysis in the revision, including:

• Visualization of how varying the similarity threshold affects selected samples,
• Analysis of attention weights during enhancement,
• Measurement of distance between original, enhanced, and hyper-modality embeddings. This will help demonstrate how hyper-modality differs meaningfully from raw or enhanced representations and why that distinction is important for sentiment understanding.

We deeply appreciate your thoughtful feedback, which has significantly improved the clarity and depth of our work. We are enthusiastic about further enhancing our manuscript based on your suggestions and thank you again for your constructive review.

We sincerely thank Reviewer QBVh for the time and effort invested in reviewing our paper, as well as for the constructive feedback provided. We are particularly encouraged that you found our approach both intuitive and original. Below, we address each of your comments in detail.

Weaknesses & Questions:

W1: Clarification of Components Based on Prior Work

Thank you for pointing out the need to clarify which components of our HME build on prior work. We agree that distinguishing the novel aspects of our approach is important for transparency. In response, we will add more details to the Appendix—such as a clearer description of frameworks like Perceiver—to better contextualize our contributions. This will help readers more easily identify what is new in our HME.

Q1: On Existing Datasets with Naturally Missing Modalities

We appreciate your insightful question regarding datasets that contain naturally missing modalities. To the best of our knowledge, such multimodal sentiment datasets are currently unavailable. As a result, the standard practice across the literature—including the baseline methods we compare against—is to simulate missing modalities by selectively dropping them from fully observed datasets. One of the key distinctions of our HME, is that it does not rely on the ground-truth representations of missing modalities during simulation. This design choice aims to more closely reflect real-world conditions and enhance the practical relevance of our method.

Q2: Motivation for the Fixed Missing Protocol

Thank you for raising the question regarding the fixed missing protocol. This is not the first time such a protocol has been used. Notable examples include baseline models like DiCMoR (ICCV 2023), IMDer (NeurIPS 2023), and LNLN (NeurIPS 2024). The motivation behind the fixed missing setting is to simulate realistic scenarios where specific modalities may be consistently unavailable—for example, when an audio sensor fails and only textual and visual information is recorded. Evaluating under this protocol helps better understand how models perform under modality-specific failures that could occur in deployment environments.

W2: Clarification on Baseline Results

We also appreciate your concern regarding the reproducibility and optimization of baseline results. In some cases, we were unable to directly compare with results reported in prior work under identical settings—either because the specific missing modality protocol was not used, or the results for certain datasets (such as IEMOCAP) were not reported for those conditions. In such cases, we carefully followed the official implementations and experimental guidelines provided by the original authors to ensure fair and meaningful comparisons. The implementation details are shown in Lines 459-464 in appendix: "To identify the hyper-parameter of the re-implemented baselines, we adhered to the hyper-parameter guidelines provided in the official implementations of these models. For models like IMDer and DiCMoR, which included pre-trained weights, we used the publicly available checkpoints and configurations. For baselines without available hyper-parameters for specific datasets, we aligned their hyper-parameters with HME’s settings where possible. When unique parameters were required, we referred to their recommended settings from other publicly available implementations."

Once again, we are grateful for your thoughtful feedback, which has helped us further improve the clarity and rigor of our work.

We sincerely thank Reviewer 5Y4d for the time and thoughtful feedback provided. We are encouraged by your recognition that our work addresses a practical and important problem with a novel cross-sample enhancement strategy. Below, we respond point-by-point to your concerns and suggestions.

Weaknesses & Questions:

W1&Q1: Comparison with State-of-the-Art (SOTA) Methods

Thank you for highlighting the need for clarification. Our baseline systems were selected from the latest state-of-the-art methods specifically designed for multimodal sentiment analysis under missing modality scenarios. These include GCNet (TPAMI 2023), MPLMM (ACL 2024), DiCMoR (ICCV 2023), IMDer (NeurIPS 2023), and LNLN (NeurIPS 2024). We will clearly state this selection criterion in the revised manuscript to avoid confusion and better contextualize our comparisons.

W2&Q2: Generalizability and Plug-and-Play Applicability of HME

We appreciate your suggestion regarding the plug-and-play applicability of HME. This inspired us to further evaluate the modularity of our HME framework. To demonstrate its generality, we integrated our Hyper-Modality Representation Generation (HMG) module into the MPLMM architecture and evaluated it under random missing protocols on the MOSI dataset. As shown in Tables 1 and 2 below, incorporating HMG improved performance in most situations, with average accuracy and F1 gains of 1.2 and 1.8 across various missing rates. These results demonstrate the plug-and-play compatibility of HME with existing architectures and confirm its general effectiveness. We will include these findings in our revised paper.

Table 1: Accuracy performance on MOSI with random missing protocol

Table 2: F1 performance on MOSI with random missing protocol

Q3: Robustness of Variance-Based Fusion

We conducted ablation studies to validate the contribution of the variance-based uncertainty fusion module. Specifically, we removed the uncertainty weighting (denoted as w/o $U_m$​) and observed performance drops across all datasets and settings, which is detailed in Tables 6 and 7 in appendix, confirming that our uncertainty estimates improve the model’s robustness in handling missing modalities.

W3&W4&Q4: Hyperparameter Sensitivity and Generalization to New Datasets

Thank you for pointing out the concern regarding hyperparameter sensitivity and potential challenges in generalizing to new datasets. Inspired by your suggestion, we conducted additional experiments on two external datasets—UR-FUNNY and MUStARD—using the same hyperparameters originally tuned on MOSI. As reported in Tables 3 through 6 (to be included in the revised manuscript), HME consistently outperformed the MPLMM baseline across various missing modality settings, even without re-tuning. These results suggest that the proposed hyperparameters generalize reasonably well across datasets, demonstrating both the scalability and robustness of our method.

Table 3: Accuracy performance on MUStARD with fixed missing protocol

Table 4: F1 performance on MUStARD with fixed missing protocol

Table 5: Accuracy performance on UR-FUNNY with fixed missing protocol

Table 6: F1 performance on UR-FUNNY with fixed missing protocol

Q5: Data Requirements for Sample Retrieval and Hyperparameter Tuning

Thank you for raising this important question. We address it from two perspectives:

• Relevant Sample Selection: The number of semantically similar samples selected by the HME is closely tied to the batch size used during training. To provide a concrete analysis, we examined this on the MOSI dataset under a missing rate (MR) of 0.3. The Table 7 below summarizes the average number of selected relevant samples (with cosine similarity > 0.9) and the corresponding model performance. Even with a batch size of 64, HME is able to retrieve nearly 8 highly similar samples on average, which already results in competitive performance. At a batch size of 128, HME outperforms all baselines, and with 256, further gains are observed. These results demonstrate that HME does not require a large batch size to find meaningful enhancements—moderate-sized batches suffice.
• Hyperparameter Tuning: Hyperparameter tuning is performed using the validation set. Taking MOSI as an example, the validation set contains 229 samples. With a batch size of 128 during validation, the effective tuning is conducted on batches of 128. We applied grid search for key hyperparameters using this relatively small validation set. Despite the limited size, the resulting configuration generalized well across both validation and test scenarios, as shown in our main experiments. In conclusion, HME is able to identify relevant samples and perform effective tuning without requiring large-scale data. This aligns well with real-world constraints and supports the practicality of our HME.

Table 7: Performance and average number of selected samples of HME in MOSI with MR=0.3

Q6: Integration with More Tightly Coupled MSA Frameworks

We appreciate your references to more tightly integrated multimodal frameworks (e.g., DeepMLF, MSP-Podcast Challenge). HME can be flexibly incorporated into these frameworks in three ways:

1. Representation-Level Integration: HME can be applied after unimodal encoders to enhance feature robustness before fusion. Our integration with MPLMM already validates this design.
2. Fusion-Level Integration: Many tightly coupled models still treat all modalities equally. Our uncertainty-aware fusion can dynamically adjust weights based on modality reliability, which is especially useful given the noisier nature of audio and video compared to text.
3. Broader Applicability to Missing Modalities: These frameworks often assume all modalities are available. HME enables them to operate in real-world missing-modality scenarios, extending their practical utility. We will elaborate on these integration strategies and provide implementation guidance in the updated manuscript.

Thank you once again for your constructive feedback. Your comments have helped us strengthen both the clarity and generalizability of our work. We hope our responses address your concerns and demonstrate the potential of HME to serve as a robust and flexible solution in real-world scenarios.

We sincerely thank Reviewer Robh for the time and effort dedicated to reviewing our paper and for the insightful comments and constructive suggestions. We are encouraged by the positive feedback, especially the recognition that our method is "technically sound and well-motivated." Below, we address the concerns and questions raised in detail.

Weaknesses & Questions:

W1&Q2: Sensitivity to Batch Composition and Size

We appreciate your observation regarding the model’s sensitivity to batch composition and size. To analyze this, we evaluated the performance of HME under different batch sizes ({64, 128, 256}) on the MOSI dataset using the random missing modality protocol. The results in Table 1 and Table 2 below show that although performance slightly varies with batch size, HME consistently outperforms baseline methods. Specifically, as batch size increases, both accuracy and F1-score improve, suggesting that more diverse batches help identify better semantic neighbors.

Table 1: Accuracy performance on MOSI with random missing protocol

Table 2: F1 performance on MOSI with random missing protocol

These experiments will be added to the appendix to clarify this sensitivity.

W2: Scalability and Pairwise Similarity Overhead

You're absolutely right that computing pairwise similarity within the batch introduces a computational overhead, especially as batch size grows. While larger batches improve the chance of finding semantically similar representations, they also increase computation. We acknowledge this trade-off and plan to explore more scalable solutions in future work, such as approximate neighbor search or memory-efficient batch construction strategies.

W3: Overloaded Mathematical Presentation

Thank you for pointing out the readability issue in Sections 3.3 and 3.4. We will revise the presentation by simplifying the notation and separating key equations from the main text for better clarity. For instance, auxiliary variables such as $\sigma + \epsilon$ will be redefined or renamed to reduce reading load for readers.

Q1: Risk of Negative Enhancement

We greatly appreciate this insightful observation. Indeed, as you pointed out, semantically similar but sentimentally dissimilar samples within a batch can introduce conflicting signals—what is referred to as “negative enhancement.” In Section A.2.4 and Table 10 of the appendix, we conduct a detailed analysis of this issue. Empirically, we observed that around 12.9% of selected neighbors in a batch could have sentiment labels opposite to the target sample, which may lead to unintended distortion of the enhanced representation.

To mitigate this, (1) we implement a similarity threshold during the neighbor selection phase. Specifically, only samples whose feature similarity exceeds a threshold are considered valid neighbors. This helps reduce the likelihood of injecting misleading information. When the similarity falls below this threshold, the model either defaults to using the average batch representation or excludes the neighbor entirely. In addition, (2) the VIB module plays a crucial role in further suppressing noise introduced by unreliable enhancements. By promoting compressed and task-relevant representations, the VIB mechanism helps filter out semantic inconsistencies and ensures that the fused features remain robust. We have observed that incorporating this strategy, especially with the VIB component, improves performance across datasets and missing-modality scenarios (as shown in Tables 6 and 7 of the appendix). We will revise the main manuscript to explicitly describe this mechanism and include additional analysis to highlight its effectiveness.

Furthermore, inspired by frameworks such as MoCo, we are exploring the use of a memory bank to retrieve reliable neighbors from a broader pool of samples beyond a single batch. This approach could further reduce the risk of negative enhancement and is a promising direction for future work.

Q3: Averaging Modality Representations When No Good Matches Exist

Thank you for this excellent question. In scenarios where no high-similarity representations are found or when a modality is entirely missing, we fall back on averaging all available representations of that modality within the batch. While these samples may come from different classes, this fallback provides a modality-level common representation that helps stabilize performance and avoid drastic degradation. Furthermore, VIB plays a crucial role in reducing irrelevant or noisy information during fusion, helping minimize the negative impact of such averaging.

Q4: Parameter Count Comparison

We appreciate the suggestion to report model complexity. As shown in Table 3 below, we report the parameter count and runtime comparison on the MOSI dataset. HME has approximately 118M parameters—slightly more than DiCMoR (113M), but fewer than most other baselines. Importantly, HME is also efficient in runtime. HME requires only 684 seconds to train, significantly less than DiCMoR’s 12,792 seconds. This reflects the practical efficiency of our approach despite the enhanced representational capacity.

Table 3:Parameter count comparison on MOSI dataset

Once again, we sincerely thank the reviewer for the thoughtful and constructive feedback. We are committed to improving the clarity, robustness, and scalability of HME, and we will incorporate all the suggested revisions in our updated manuscript and supplementary material.