Learning Optimal Multimodal Information Bottleneck Representations

Experimental Designs Or Analyses:

Q1. Thank you for this insightful observation regarding synergistic interactions between modalities. In response, we conducted additional experiments using synthetic data with two modalities ($x_1,x_2$), where $x_1=[a_0;b_0]$, $x_2=[a_1;b_1]$, and $y=\Delta(a_0^Ta_1>0)$. All $a_i$ and $b_i$ are sampled from the standard Gaussian, yielding 10,000 sample pairs. Theoretically, in this setting, the TRBs would be unable to distinguish between $a_i$ and $b_i$, leading to the encoder to randomly incorporate information. However, $L_{OMF}$ of the OMF block can capture the inter-modal synergic interactions and inform the encoders to extract task-relevant information. To validate this, we tested our method in two configurations: $L_{OMF}$ is either involved (w-$L_{OMF}$) or not (wo-$L_{OMF}$) in optimizing the modality encoders. For comparison, we also evaluated a single-modality baseline (Single) and the union of the two modalities (Union). The table below shows that w-$L_{OMF}$ achieves the highest accuracy, followed by Union and wo-$L_{OMF}$, while Single performs near random guessing. These results align with our expectations and demonstrate OMF block's effectiveness in capturing inter-modal synergetic interactions.

Q2. As you suggested, we increased the training epochs from 100 to 1,600 and observed that our method slightly improved, with macro-accuracy increasing from 63.6% to 65.4%. We agree that introducing KS or VGGSound datasets can help to more accurately evaluate our method. However, due to the limited rebuttal period and the very large data sizes involved, we plan to leave these results in the revised manuscript later.

Questions For Authors:

Q1. Thank you for pointing us to these recent methods. Since OGM-GE is originally designed for the classification task, we have included it as a benchmark for CREMA-D only (note that CMU-MOSI is a regression task, and anomalous tissue detection is an SVDD-based unsupervised task). Specifically, OGM-GE achieves a 61.0% accuracy on CREMA-D. Regarding MLA, we regret that its codes have been withdrawn, preventing us from adding it as a benchmark. Nonetheless, we have now discussed both methods in the Related Work and Experiment sections of this revision.

Q2. Concatenation can be viewed as a generalized addition for combining the representation $z$ and noise $e$. To see this point, for concatenation, we have $[z^T,e^T] [\matrix{ W_1 \cr W_2}] =z^TW_1+e^TW_2$, which reduces to additive form $(z+e)^TW$ when $W_1= W_2$. Notably, conditional GAN (Mirza et al. 2014) also integrates noise vectors via concatenation for controlled generation. Moreover, using concatenation allows us to replace $e$ with the fused MIB $\xi$ during main training without modifying TRB's network architecture, thus offering more flexible cross-modal interactions compared to addition. Our ablation study on the CREMA-D dataset further confirms that the concatenation of $e$ (or $\xi$) with $z$ slightly outperforms a simple additive combination (macro-accuracy: 63.6% vs. 62.4%).

Other Comments Or Suggestions:

Properties 1 has been removed from the main text.

Weaknesses

W1. Thank you for reminding us of this important point. Robustness to estimation errors: MINE is a theoretically validated estimator with strong consistency, which is achieved by optimizing the Donsker-Varadhan (DV) representation, a lower bound of the true MI (Belghazi et al., 2018). The optimal function that tightens the DV bound can be approximated using neural networks owing to universal approximation theorems, ensuring MINE's convergence to the true MI as sample size grows. Our experiments use datasets with relatively large sample sizes (e.g., Pathological Tissue (10,129)), the estimation error thus is expected to be minor. To further mitigate batch-induced bias, we apply exponential moving averaging on gradients during MINE’s optimization. Additionally, in its original study, MINE was shown to accurately estimate mutual information for generating superior single-modal IB for MNIST classification, empirically demonstrating its reliability. Finally, the theoretical bounds $M_u$ and $M_l$ are designed conservatively to tolerate estimation errors. For instance, $M_u$ is set to $\frac{1}{3 (H(v_1)+H(v_2)-I(v_1;v_2))}$, which intentionally tightens the upper bound to ensure $\beta$ remains within a safe range even if $H(v)$or $I(v_i;v_j)$ are slightly misestimated, thereby safeguarding convergence to optimal MIB. Scalability: MINE scales linearly with data dimensionality and sample size, as shown in its original work. Moreover, its computational cost is amortized since it is only used once to estimate $M_l$ and $M_u$ prior to training.

W2. We apologize for the confusion. Actually, OMIB adopts the late-fusion strategy (the OMF block) similar to L-MIB (Mai et al. 2022), where unimodal representations are first condensed to reduce noises via variational IB encoders before fusion using a light CAN. This contrasts with the heavier early fusion occurring in the full information space. To empirically verify OMF's scalability to large-scale datasets, we generated six large synthetic datasets as in SIM-I–III and measured OMF's training and inference time per epoch (see the table below). The results indicate that OMF scales linearly with large-scale datasets.

W3. Since most MIB studies consider up to three modalities, we followed this routine. To address your concern about modality scalability, we conducted an additional experiment on synthetic datasets with up to five modalities. Each modality consists of a shared task-relevant, a unique task-relevant, and a unique task-irrelevant component. To isolate the impact of the number of modalities, the dataset size was fixed at $10^5$. The observed time costs (see Table below) scale approximately quadratically with $M$, consistent with the expected $\mathcal{O}(M^2)$ complexity.

W4. Thank you for this excellent comment, which coincides with the most significant challenge of deep MIB learning proposed by Shwartz-Ziv and LeCun (2023). Our primary goal is to establish the theoretical achievability of optimal MIB under the classical MIB paradigm, where downstream task labels are available. As noted by Tian et al. (2020), the optimal MIB inherently depends on the specific task, since what is relevant for one task may be irrelevant for another. So optimal MIB may not be well defined without task labels in the first place. In our formulation, both shared and Modality-Specific Task-Relevant (MSTR) contents are considered, which further complicates the extension to label-free settings. Regarding self-supervised learning (SSL), while recent methods (e.g., DISTANGLEDSSL) have attempted to learn optimal MIB representations without labels, they typically rely on the strong MultiView assumption (Sridharan et al. 2008) that neglects MSTR. When this assumption is violated, MSTR can be excluded. Although this issue can be mitigated by adding regularization to increase the MI between the representations and inputs, the achievability of task-specific optimal MIB is agnostic. Thus, the guarantee of achieving optimal MIB with SSL in the presence of MSTR could be unattainable. For semi-supervised learning, a potential approach is to train OMIB on the available labeled data and then propagate labels to unlabeled data, with regularization such as a prior label distribution from labeled data to enhance generalization. However, the achievability of optimal MIB is not guaranteed either.

Essential References Not Discussed

We also thank you for pointing us to the DISTANGLEDSSL paper, which offers valuable insights into the SSL-based MIB. We will include and discuss it in the related work section of the revised version.

Questions For Authors

Q1. This is a good point. To mitigate potential instability arising from extreme KL-divergence ratio ($KL_r$), we adopt several strategies. First, the raw $KL_r$ is not directly used; instead, the weight $r$ is computed as $1-tahn(\cdot)$ and bounded, thus preventing extreme values. Since the $tahn$ function is smooth and saturates, $r$ will not change abruptly in case of significant information imbalance (e.g., very large $KL_r$), thus promoting the training stability. Second, we will add a small constant $\epsilon$ to the denominator in the computation of $KL_r$ to avoid division by $0$ and enhance numeric stability. Third, the batch-averaged KL ratio (i.e. $\frac{1}{N}\sum_{n=1}^N r_n$) is used to compute $r$, thus smoothing out sample-level fluctuations of information ratio. Finally, Proposition 1 guarantees the convergence to the optimal MIB under bounded $r$ and $\beta$, regardless of the information imbalance. Empirical results from our synthetic experiments (SIM-I and -II), where one modality is designed to contain significantly more or less information than the other, further demonstrate this point.

Q2. Thanks for this insightful comment. While CAN does not include an explicit contrastive loss, our training objective implicitly enforces diversity. As proved in Lemma 1, CAN's training objective function (Equation 17) ensures that both consistent (shared) and diverse (modality-specific) task-relevant contents are captured in the MIB upon convergence. In particular, the redundancy penalty terms ($I(\xi;z_i)$) in the objective function prioritize diverse, informative content as modality‐specific features incur less redundancy compared to shared features. Empirically, our synthetic experiments (Table 2) show that our method significantly surpasses MIB that exclusively contains shared& task-relevant content, and achieves performance comparable to the authentic optimal MIB. These results confirm the effectiveness of our OMIB framework in accounting for information diversity. We acknowledge that incorporating a contrastive learning term could further promote diversity. However, adding such a term would complicate the theoretical analysis of achieving the optimal MIB and might disrupt the delicate balance between minimizing redundancy and maximizing task relevance established by our current objective. We consider this an intriguing direction for future work, particularly in exploring whether contrastive loss and redundancy penalty are functionally equivalent in enforcing diversity.

Q3. Thank you for highlighting this practical concern. The primary goal of this work is to establish a rigorous theoretical foundation for achieving optimal MIB under the classical MIB paradigm. To allow tractable analysis, we adopt simplified assumptions, including the availability of all modalities, which admittedly do not fully capture real-world complexities. However, these assumptions align with the spirit of Ali Rahimi’s NIPS 2017 Test of Time Award remark—“Simple experiments, simple theorems are the building blocks that help us understand more complicated systems”.

Your suggestion of incorporating additional mechanisms for handling missing modalities is valuable. One promising approach would be to use modality-complete data to train an auxiliary VAE that maps one modality's observations to the variational parameters of the other. In cases of modality-incomplete data, the available modality could then approximate the variational representations of the missing one via the reparameterization trick prior to CAN-mediated fusion. However, integrating this component would alter the framework's architecture and training objectives and compromise its theoretical guarantees, deviating from our study's original focus. Rather, these extensions constitute our de novo future work with a more practical orientation.

Q4. We conducted an additional experiment using the MM-IMDb dataset—a challenging text-visual dataset with severe class imbalance. Specifically, it consists of 25,959 sample pairs across 23 movie genres, with the largest class containing 13,967 samples and the smallest 338, representing a 41-fold imbalance. A stratified split was applied to form training (60%), validation (10%), and testing (30%) sets. We evaluated our method and nine benchmark methods using the macro F1-score (see Table below). The results demonstrate that our method outperforms the benchmarks in this more challenging setting.