InfMasking: Unleashing Synergistic Information by Contrastive Multimodal Interactions

Response Q.1 and L.1: The phenomenon observed in the ablation study (Figure 2(a)), where more masked views do not necessarily lead to better performance, can be attributed to the balance between capturing synergistic information and avoiding representational collapse. As described in the paper (Page 9, Section 5), the InfMasking approach relies on randomly masking a significant portion of features from each modality during fusion to create diverse synergistic patterns. However, increasing the number of masked views excessively may lead to insufficient information retention, causing the model to struggle with learning robust representations.

Specifically, the paper notes that the optimal performance on the MIMIC and MOSI datasets is achieved at a masking ratio of 0.5 or 0.6 (Page 9, Figure 2(b)). Beyond this range, performance declines because excessive masking can result in representational collapse, where the model fails to retain enough complementary information across modalities to form generalizable representations. With too many masked views, the model may overly rely on limited subsets of features, reducing the diversity of synergistic interactions and hindering the alignment of masked and unmasked representations through mutual information maximization.

Thus, the key is to maintain a controlled number of masked views to ensure sufficient information is preserved while still encouraging the model to infer complementary, synergistic information across modalities. This balance is critical for enhancing the robustness and generalizability of the learned representations, as discussed in Section 3.2 (Page 5).

We can draw from [1] and add uniform loss to solve this representational collapse. In the revision version, we further analyze this issue and conduct experiments for comparison.

Response Q.2 and L.2: Measuring multimodal interactions as defined by PID is challenging since it involves estimating information-theoretic measures. The problem is even harder for high-dimensional data distributions. Currently, there is no well-established metric for measurement. We referenced the ICLR paper Comm[2] and conducted synthetic experiments on trifeature datasets in Section 4.1 to validate that our method more effectively learns synergistic information.

Response Q.3 and L.3: Data augmentation plays a critical role in our unsupervised pre-training method, as outlined in Section 2. For Contrastive Multimodal Interactions (CoMM), data augmentation is essential to create two distinct yet mutually supervised multimodal views, enabling effective contrastive learning. The authors chose two augmentation methods, likely to align with CoMM for fair comparison.

[1] Zhang, Q., Wang, Y., and Wang, Y. How mask matters: Towards theoretical understandings of masked autoencoders.Advances in Neural Information Processing Systems, 35: 27127–27139, 2022b. 1, 3, 4, 7, 8

[2] Benoit Dufumier, Javiera Castillo-Navarro, Devis Tuia, and Jean-Philippe Thiran. What to align in multimodal contrastive learning? In International Conference on Learning Representations, 2025.

Thank you for acknowledging our rebuttal. We truly hope our detailed response is helpful in clarifying our work and addressing your concerns. We would welcome any further discussion you might suggest.

Thank you for your feedback and for raising the score. We appreciate your thorough review. Regarding your concern about Q3&L3, we agree that exploring additional data augmentation techniques could provide valuable insights into the method’s scalability and performance upper bound. As noted in [1], additional stronger data augmentation techniques have been shown to improve the performance of contrastive learning. To address your suggestion, we are happy to conduct ablation experiments to evaluate the impact of additional augmentation strategies on our method’s performance. These experiments will be included in the revised manuscript to further validate the approach and provide a clearer understanding of its potential.

[1] “Contrastive Learning With Stronger Augmentations,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 5, pp. 5549-5560, 1 May 2023.

W.1 The method requires k masking operations which might resulting in higher computational cost than traditional approaches. This might not suitable for large-scale datasets.

Response W.1: As shown in Figure 1, our masking approach targets the features of each modality before fusion, rather than the raw input, minimizing computational overhead. The repeated sampling process adds negligible cost. The primary computational increase stems from fusing masked views, using a lightweight fusion network with a single Transformer layer. For large-scale datasets like V&T CP (3 modalities) and IMDB (2 modalities), we apply a masking ratio of 0.8, retaining 20% of tokens unmasked. With 5 and 4 masked views, respectively, the computational impact remains minimal, as sampling can be parallelized during fusion.

Furthermore, our method serves as an unsupervised pre-training approach. During downstream task inference and fine-tuning, it introduces no additional computational overhead compared to standard methods.

W.2 This random masking method has shortcomings. It is difficult to take into account the fact that different data sets have different characteristics.

Response W.2: The random masking method has limitations, as it struggles to adapt to the unique characteristics of different data sets. In contrast, our infinite mask strategy overcomes this by leveraging the Infmasking loss to estimate mean and variance, effectively capturing the distinct properties of various data sets without requiring specific adjustments for each.

W.3 There is a problem of hyperparameter sensitivity. Different data sets may require completely different parameter settings. There is also a lack of an automatic parameter adjustment mechanism.

Response W.3: Small multimodal datasets like MOSI are highly sensitive to hyperparameters, such as the seed, resulting in significant performance variance. However, larger datasets like IMDB and V&T CP exhibit less sensitivity to hyperparameters like the seed. As shown in Table 6, a masking ratio of 0.7 or 0.8 typically performs well, with values between 4 and 6 being viable.

W.4 The method primarily benefits tasks where synergistic information is critical. For scenarios where redundant information is more valuable, InfMasking may provide limited improvements, lacking adaptive mechanisms to adjust to different task characteristics and information interaction patterns.

Response W.4: Liang et al. [1] conducted an extensive human evaluation study across 30 multimodal datasets to identify the dominant information-theoretic interactions (Table 1). Their results indicate that redundancy is prevalent in many VQA tasks, but they identified 8 datasets—including IRFL, MM-IMDb, and MAGIC BRUSH—where synergistic interactions dominate.

Our method is a general-purpose pretraining approach. Currently, determining dataset-specific interaction patterns relies on human evaluation. As shown in Table 2, in datasets like MIMIC [1] where redundancy predominates, InfMasking yields limited improvements but does not hinder model performance. For known tasks, we can design a mixture-of-experts approach to adaptively handle varying information interaction types. This method can serve as our future research work.

Q.1 How robust is the Gaussian approximation assumption in practice?

The method relies on assuming that masked representations follow a Gaussian distribution to derive the computationally tractable lower bound. However, in real multimodal scenarios, the distribution of fused features after masking could be highly complex, multimodal, or skewed. What happens when this assumption breaks down? Have the authors tested the sensitivity of their approximation quality across different data distributions, and how does deviation from Gaussianity affect the theoretical guarantees and practical performance?

Response Q.1: This assumption is well-founded for two principal reasons. First, the masked embeddings tend to cluster around a central value in the embedding space, as they all inherently reflect aspects of the synergist’s semantic nature. Second, the variance observed across feature dimensions can be interpreted as a representation of semantic differentiation in the ambient space. In the revised version, we have included visualizations to support this argument.

In practice, the Gaussian assumption holds reasonably well for high-dimensional representations after feature fusion (e.g., via a Transformer), as the aggregated features from multiple modalities tend to exhibit central limit theorem-like behavior due to the averaging effects of fusion. Our experiments on diverse real-world datasets, such as those in MultiBench (Section 4.2), spanning domains like healthcare, robotics, and multimedia (e.g., MIMIC, MOSI), demonstrate that InfMasking achieves state-of-the-art performance across seven benchmarks. This suggests that the Gaussian approximation is sufficiently robust for a wide range of practical multimodal scenarios, even when the underlying data distributions may deviate from strict Gaussianity.

We sincerely appreciate the reviewer's time and consideration, and we are happy to provide additional details or revisions if needed. There are only two days left until the deadline.

W.1 Synergy is vaguely described as "complementary information emerged only when … combined" without formal measures beyond PID. No real-world examples illustrate tasks where synergy is critical and redundancy/uniqueness is insufficient. One example that comes to my mind is hateful meme detection. A neutral image combined with neutral text generates harmful content that neither modality conveys alone. Including such real-world examples would strengthen the motivation.

Response: Thank you for this valuable suggestion. We agree that concrete real-world examples would significantly strengthen our motivation for studying synergy. In the revision, we will add the hateful meme detection example you mentioned in the introduction, as it perfectly illustrates how synergy emerges when seemingly neutral modalities combine to create harmful content that neither conveys alone.

Additionally, we will include the following content to provide empirical evidence for the importance:

"Liang et al. [1] conducted a comprehensive human evaluation study across 30 multimodal datasets to determine the dominant information-theoretic interactions (Table 1). While their findings show that redundancy predominates in many VQA tasks, they identified 8 datasets—including IRFL, MM-IMDb, and MAGIC BRUSH—where synergy is the dominant interaction. This empirical evidence demonstrates that synergy is not merely a theoretical concept but a measurable phenomenon critical for understanding multimodal learning in real applications.

W.2 In section 3.2, while the derived lower bound makes the loss calculation more tractable, the training still involves repeatedly sampling and fusing masked views. This approach may face scalability challenges with three or more modalities, where the combinatorial complexity increases substantially.

Response: As illustrated in Figure 1, our masking approach does not mask the raw input of each modality, but rather masks the features of each modality before fusion. The repeated sampling process incurs negligible computational cost. The only computational overhead comes from fusing the masked views, where the fusion network typically consists of a single Transformer layer. For large-scale datasets like V&T CP (3 modalities) and IMDB (2 modalities), we set the masking ratio to 0.8, meaning only 20% of tokens remain unmasked. With 5 and 4 masked views, respectively, the computational increase is minimal, and each sampling can be processed in parallel during fusion.

Furthermore, our method serves as an unsupervised pre-training approach. During downstream task inference and fine-tuning, it introduces no additional computational overhead compared to standard methods.

W.3 In Table 5, it’s interesting that setting to zero increases redundancy and uniqueness while reducing synergy. A brief discussion of why this behavior occurs might offer additional insight.

Response: $R$ represents redundancy, $U_i$ denotes uniqueness for view $i$, and $S$ indicates synergy.

Setting $\lambda_3 = 0$ eliminates the term $\hat{I}_{\text{NCE}}(Z', Z'')$

from the loss function. This results in an increase in $U$ but a reduction in synergy $S$ compared to including this term. Theoretically, this term should enhance redundancy, uniqueness, and synergy simultaneously. However, in practice, uniqueness decreases when this term is included. This behavior likely occurs because the $\hat{I}_{\text{NCE}}(Z', Z'')$ term prioritizes learning redundancy and synergy across views, which can constrain the model's ability to capture view-specific information (uniqueness). When this term is removed, the model focuses more on view-specific features, boosting uniqueness at the expense of synergy.

W.4 The evaluation lacks comparisons masking baselines (e.g., MultiMAE [1]) or methods explicitly modeling interactions (e.g., attention bottlenecks [2]).

Response:

(1) For the purpose of comparison with MultiMAE, under the same experimental conditions, we conducted experiments using the loss function from MultiMAE on multiple 2-modal datasets. The average results across five seeds (42–46) are as follows:

(2) Attention bottlenecks is a multimodal fusion network that can be directly applied to CoMM and our method as fusion networks. For a fairer comparison with its baseline, we only use a simple Transformer for fusion.

[1] Liang, P. P., Goindani, A., Chafekar, T., Mathur, L., Yu, H., Salakhutdinov, R., & Morency, L. P. (2024). Hemm: Holistic evaluation of multimodal foundation models, NeurIPS 2024

We sincerely appreciate the reviewer's time and consideration, and we are happy to provide additional details or revisions if needed. There are only two days left until the deadline.

Dear Reviewer,

We sincerely appreciate the time and effort you have dedicated to reviewing our submission for NeurIPS 2025. Your feedback is incredibly valuable to us and plays a critical role in refining our work.

We are grateful to note that Reviewer Z2T2 has kindly raised their score, and Reviewer P89E has maintained their positive score. Your comments remain of utmost importance to us, and we are eagerly awaiting your response to our rebuttal.

As a fellow reviewer for NeurIPS 2025, I deeply understand the demands on your time and greatly admire the care you bring to this process. Your thoughtful input would be immensely appreciated, and we look forward to your valuable feedback.

Thank you once again for your time and consideration.

Best regards,

Authors

Response W.1 and Q.2: Small multimodal datasets like MOIS are highly sensitive to hyperparameters, such as the seed, resulting in significant performance variance. However, larger datasets like IMDB and V&T CP exhibit less sensitivity to hyperparameters like the seed. As shown in Table 6, a masking ratio of 0.7 or 0.8 typically performs well, with values between 4 and 6 being viable.

Response W.2 and Q.3: Measuring multimodal interactions as defined by PID is challenging since it involves estimating information-theoretic measures. The problem is even harder for high-dimensional data distributions. Currently, there is no well-established metric for measurement. We referenced the ICLR paper Comm[1] and conducted synthetic experiments on trifeature datasets in Section 4.1 to validate that our method more effectively learns synergistic information.

Response W.3 and Q.1:

(1) Although MAE's reconstructive approach can also learn collaborative information, downstream task prediction considers all tokens, leading to a gap between pre-training and downstream tasks. InfMasking's contrastive objective aligns unmasked fused representations with masked ones via mutual information maximization to encode synergistic information.

(2) MAE's reconstructive approach typically converges slowly, with high pre-training costs. We adopt the infinite masking strategy and derive an InfMasking loss to approximate the calculation of this loss function.

(3) Our masking approach does not mask the raw input of each modality but rather masks the features of each modality before fusion.

For the purpose of comparison with MAE, under the same experimental conditions, we conducted experiments using the loss function from MAE on multiple 2 modalities datasets. The average results across five seeds (42–46) are as follows:

Response Q.4: As illustrated in Figure 1, our masking approach does not mask the raw input of each modality, but rather masks the features of each modality before fusion. The repeated sampling process incurs negligible computational cost. The only computational overhead comes from fusing the masked views, where the fusion network typically consists of a single Transformer layer. For large-scale datasets like V&T CP (3 modalities) and IMDB (2 modalities), we set the masking ratio to 0.8, meaning only 20% of tokens remain unmasked. With 5 and 4 masked views respectively, the computational increase is minimal, and each sampling can be processed in parallel during fusion.

Response Q.5: In the 3 modalities experiments on the UR-FUNNY dataset, we conducted a zeroing operation on each individual modality (i.e., vision, text, and audio) respectively, and conducted experiments on both the CoMM and InfMasking models. The average results across five seeds (42–46) are as follows:

[1] Benoit Dufumier, Javiera Castillo-Navarro, Devis Tuia, and Jean-Philippe Thiran. What to align in multimodal contrastive learning? In International Conference on Learning Representations, 2025.

We sincerely appreciate the reviewer's time and consideration, and we are happy to provide additional details or revisions if needed. There are only two days left until the deadline.

Dear Reviewer,

We sincerely appreciate the time and effort you have dedicated to reviewing our submission for NeurIPS 2025. Your feedback is incredibly valuable to us and plays a critical role in refining our work.

We are grateful to note that Reviewer Z2T2 has kindly raised their score, and Reviewer P89E has maintained their positive score. Your comments remain of utmost importance to us, and we are eagerly awaiting your response to our rebuttal.

As a fellow reviewer for NeurIPS 2025, I deeply understand the demands on your time and greatly admire the care you bring to this process. Your thoughtful input would be immensely appreciated, and we look forward to your valuable feedback. Thank you once again for your time and consideration.

Best regards, Authors

We sincerely appreciate the reviewer's time and consideration, and we are happy to provide additional details or revisions if needed.