An Information Criterion for Controlled Disentanglement of Multimodal Data

We would like to thank the reviewer for their diligent and insightful review and their encouraging words. We have incorporated the reviewer’s comments in the updated paper, with the changes in the manuscript highlighted in magenta. We share our thoughts on the questions asked below.

How does the optimization objective for the modality-shared representation change from Eq. (1) to Eq. (3)? Why is $I(X^1;X^2)$ omitted?

The Lagrangian form of the constraint optimization problem in Eq. (1) has the formula $\tilde{L} = I(Z^1;X^1|X^2) + \tilde{\beta} \cdot (I(X^1;X^2) - I(Z^1;X^2))$ Since we are optimizing for the representation $Z^1$, with given $X^1$ and $X^2$, the term $I(X^1;X^2)$ is a fixed constant irrelevant to $Z^1$ and can be omitted from the objective, i.e. $\arg \min_{Z^1} \tilde{L} = \arg \min_{Z^1} I(Z^1;X^1|X^2) - \tilde{\beta} \cdot I(Z^1;X^2)$ This is equivalent to the objective in Eq. (3), i.e. $\arg \min_{Z^1} \tilde{L} = \arg \max_{Z^1} L_c^1 = \arg \max_{Z^1} I(Z^1;X^2) - \beta \cdot I(Z^1;X^1|X^2)$

Additionally, the properties of the optimal representations learned by Eq. (3) are more rigorously justified in Proposition 1 and Proposition 2.

Related studies, such as CoCoNet and SimMMDG, are recommended for inclusion and comparison.

We have included CoCoNet and SimMMDG in the updated Section 4 with discussions. Also, we experiment with SimMMDG in the simulation study, with results shown in the updated Figure 4b, Figure 8, and Figure 9. DisentangledSSL outperforms SimMMDG in this setting.

Besides, although the methods in CoCoNet and SimMMDG are related to DisentangledSSL, their specific problem settings are quite different from ours. CoCoNet focuses on learning image representations via modeling multiple views of one image (i.e. L, RGB, ab). SimMMDG focuses on a domain generalization setting with supervised labels. In contrast, we focus on the self-supervised setting with general multimodal data.

Ablation studies on the impact of $\beta$ and $\lambda$ in the MultiBench datasets should be provided to demonstrate the importance of separating modality-specific and modality-shared information in real-world applications.

Thank you for your valuable suggestion. We have added an ablation study of DisentangledSSL on the MultiBench datasets to examine the impact of the hyperparameters $\beta$ and $\lambda$ in Table 5. Further, the effectiveness of separating modality-specific and modality-shared information is already reflected in the superior performance of DisentangledSSL compared to other baselines.

Some typos. 1) $Z^2$ should be in $X^2$ Eq.(6) and in the equation below Eq.(6). 2) $I(X^1;Z^1)$ should be $I(X^1; X^2)$ in Figure 3.

Thanks a lot for pointing out the typo in Equation (6). We have corrected it in the revised manuscript. However, $I(X^1; Z^1)$ is correct in Figure 3. It indicates the values in the horizontal axis and is measured for different $Z^1$.

Are there any alternative approaches for implementing the $I(Z^1; \hat{Z}_c^{1*})$ in $L_s^1$? If so, how does their performance compare to that of the orthogonal loss implementation?

That's a great point! Yes, there are alternative approaches for implementing the mutual information term $I(Z^1; \hat{Z}_c^{1*})$ in $L_s^1$. One such approach is adversarial training (eg., [1,2]), whereby an encoder and discriminator play a minimax game. However, adversarial training can be unstable and difficult to tune, which makes it less practical for some applications. Other approaches include applying measures like the Hilbert-Schmidt Independence Criterion (HSIC) [3] or Distance Correlation as regularizers to encourage statistical independence.

In contrast, we use an orthogonal loss, which is a simple and effective method to minimize the mutual information between $Z^1$ and $\hat{Z}_c^{1*}$. That said, DisentangledSSL can accommodate different implementations of each term in the loss function, allowing for easy integration of future advances in mutual information optimization.

[1] Disentangled Information Bottleneck. AAAI 2021.

[2] Learning disentangled representations via mutual information estimation. ECCV 2020.

[3] A Kernel Statistical Test of Independence. NeurIPS 2007.

Thanks for the responses. My concerns are sufficiently addressed, and I'd like to maintain my positive rating.

Thank you for your response and insightful review!

The authors have conflated the generative and inference processes in the graphical model in Figure 2.

Thank the reviewer for this suggestion. We have updated the graphical model with a separate generative process and inference process in the revised paper. Nevertheless, it has become a common practice in previous papers to combine generation and inference parts in the same graphical model when there’s little chance for confusion. The graphs are used as illustrations, not intended as formal graphical models. For example, you would see this in [4] in the context of multimodal representation learning, and [5] in the context of diffusion models.

[4] Private-Shared Disentangled Multimodal VAE for Learning of Latent Representations. CVPR 2021.

[5] Denoising Diffusion Probabilistic Models. NeurIPS 2020.

The assumption of a shared latent variable $Z_c$ existing between two modalities may not hold.

In this paper, we focus on modeling the statistical relationships between the domains rather than causal relationships. Therefore, the three graphical relations between variables proposed by the reviewer are statistically (marginally) equivalent, and are in our case represented by a single variable $Z_c$ (variability specific to a domain would be delegated to other variables). This statistical setup is not unique to our paper but commonly used in many past works that also learn disentangled representations of multimodal data, for example, [6,7,8,9,10,11]. These involve data generation processes with a single shared latent variable and modality-specific latent variables corresponding to each modality.

We emphasize that the focus and the main contribution of our paper is not to study a causal graphical model but rather on how to extract and infer the representations from the data. In particular, we address the challenging real-world setting where Minimum Necessary Information (MNI) is not attainable. This case is underexplored in prior works. We provide detailed and rigorous analysis in Sections 2.2 and 2.3 and demonstrate the efficacy of our proposed method through comprehensive experiments in Section 3.

[6] Private-Shared Disentangled Multimodal VAE for Learning of Latent Representations. CVPR 2021.

[7] Self-supervised Disentanglement of Modality-specific and Shared Factors Improves Multimodal Generative Models. Pattern Recognition 2020.

[8] Variational Interaction Information Maximization for Cross-domain Disentanglement. NeurIPS 2020.

[9] FOCAL: Contrastive Learning for Multimodal Time-Series Sensing Signals in Factorized Orthogonal Latent Space. NeurIPS 2023.

[10] Factorized Contrastive Learning: Going Beyond Multi-view Redundancy. NeurIPS 2023.

[11] The Platonic Representation Hypothesis. ICML 2024.

We are grateful to the reviewer for the time they put in to review our work. We are glad to see that they recognize several strengths in our work, including theoretical analysis based on the unattainability of MNI and comprehensive empirical evaluation across a range of synthetic and real-world multimodal datasets. Below, we share our thoughts on the questions asked and have updated the manuscript with changes highlighted in magenta.

The statement "shared information between modalities is precisely what is relevant for downstream tasks" is not universally accurate. Furthermore, "shared information" is a vague term, especially when applied across modalities, making it problematic in this context.

It appears there may be some misunderstanding. This statement is used in our paper to point out the unrealistic assumption of “multi-view redundancy” exploited by previous works (e.g., [1,2,3]) that only maximizes the mutual information between modalities. “Shared information” in these contexts indicates the information that is common and redundant to all modalities. This assumption is a limitation of these prior methods, and our work does not adopt it. Instead, we consider the general case where downstream tasks are related to both shared and modality-specific information.

[1] An information theoretic framework for multi-view learning. COLT 2008.

[2] Contrastive learning, multi-view redundancy, and linear models. Algorithmic Learning Theory 2021.

[3] Self-supervised learning from a multi-view perspective. ICLR 2020.

The modality gap is not the sole or primary cause of misalignment; other factors, such as alignment methods, distributional shifts, domain shifts, sampling biases, and more, can contribute significantly.

We agree with the reviewer that the modality gap is not the sole cause of misalignment between modalities; many factors including alignment methods, distributional shifts, domain shifts, and so on, also contribute. However, the modality gap is one of these key factors and serves as the primary motivation for our method that learns both shared and modality-specific representations. While we acknowledge that other factors can cause misalignment, addressing them (all) is not the main focus of our study here.

I suggest the authors refine their motivation to more accurately reflect the complexities of multimodal relationships, while still maintaining the core idea of disentangling shared and modality-specific information.

In response to the reviewer’s suggestion, we have expanded our introduction to more thoroughly discuss the complexities of multimodal relationships. Please find the revisions indicated in magenta color in the revised manuscript.

Thank you for your response and increased score!

First of all, we would like to thank the reviewer for their diligent and insightful review and their very encouraging words. The changes in the manuscript can be found highlighted in magenta. We share our thoughts on the questions asked below.

The author should consider Including a Pseudocode/Alg block to clearly demonstrate the training process.

We thank the reviewer for the suggestion. That indeed is a great point! We have revised the manuscript with an additional pseudocode and a separate table for notations in Appendix I and Appendix J to improve the clarity of our work.

DisentangledSSL-shared or DisentangledSSL-specific alone sometimes achieves top performance.

Please refer to the detailed response about key observations from this table in the consolidated review to all reviewers above.

Thank you for your response and insightful review!

The performance gain in Table 2 is limited.

DisentangledSSL significantly outperforms other baselines in Table 2 in most cases, especially in the RXRX19a dataset. For example, DisentangledSSL has on average 8.6% gain in top N retrieving accuracy over the best-performed baseline on RXRX19a.

Further, we note that such a retrieval task (i.e., identifying the molecule perturbation that corresponds to a certain phenotype from a large collection of molecules) is a difficult task, especially when we test on held-out unseen molecules, and it suffers from the noise and potential biases in the dataset. For example, the effect of some drugs may be insignificant, and different drugs may share similar phenotypes [3].

Therefore, even relatively small gains on these tasks are significant. This is also reflected in the results of previous papers on similar tasks (molecule-cell image retrieval) [1,2,3]. For example, in [3], the top-1% recall accuracy of unseen molecules over all molecules is 28.5%, as compared to the best-performed baseline of 27.0%.

[1] Cross-Modal Graph Contrastive Learning with Cellular Images. Advanced Science 2024.

[2] Removing Biases from Molecular Representations via Information Maximization. ICLR 2024.

[3] How Molecules Impact Cells: Unlocking Contrastive PhenoMolecular Retrieval. NeurIPS 2024.

The paper does not satisfy the page limitation.

According to the ICLR 2025 Official Author Guide (https://iclr.cc/Conferences/2025/AuthorGuide), the optional reproducibility statement (Section 6) does not count toward the page limit of 10 pages.

We thank the reviewer for their time and feedback that helped us improve the work. We believe that there are a few confusions about our experimental results that have resulted in the given rating. We have endeavored to address your concerns as concretely as possible and ask for your careful consideration of our clarifications. All of the discussions below are added to the revised manuscript in magenta.

We want to emphasize that our work is among the first to tackle the modality gap in multimodal representation learning from the perspective of the unattainability of Minimum Necessary Information (MNI). This, as we believe, is an extremely important contribution, as other methods often assume the attainability of MNI, a condition that almost never holds true in practice.

Synthetic experiments for varying degrees of MNI attainable or unattainable.

This is indeed an important question. Following the reviewer's suggestion, we evaluated our method on data with varying noise levels. As shown in Figure 11 of the revised manuscript, as the noise level decreases, the learned shared representation $\hat{Z}_c$ contains less modality-specific information under the same $\beta$, and the test accuracy on $Y_1$ and $Y_2$ decrease to close to 0.5 even with small $\beta$. Meanwhile, the frontier of test accuracy on $Y_c$ versus $Y_1$ shrinks towards the top left corner, indicating a decreasing expressivity-redundancy tradeoff and easier disentanglement. These trends align well with our theoretical analysis.

Additionally, we refer the reviewer to Figure 10 in Appendix H.1, which experiments on synthetic data with different levels of multimodal entanglement and illustrates the trade-off between expressivity and redundancy in the learned shared representations. In this context, some dimensions correspond to scenarios where MNI is attainable, while others represent cases where it is unattainable. Using such data, DisentangledSSL shows a clear pattern where the learned information plateaus at certain $\beta$ values, which aligns with the property of partially attainable MNI in this synthetic data.

Superiority of DisentangledSSL is not significant compared to baselines in Figure 4(b).

The ideal frontier for the specific information in Figure 4(b) is the one where the specific representation $\hat{Z}_s^1$ achieves low accuracy on the shared target $Y_c$ (y-axis) and high accuracy on the specific target $Y_s^1$ (x-axis). This optimal frontier corresponds to the bottom-right corner of the plot. Our method, DisentangledSSL, produces a frontier closest to this ideal, outperforming all baselines—including DMVAE, FOCAL, and JointOpt. Additionally, Figure 8 in the Appendix presents the frontier for the shared representation $\hat{Z}_c$, where DisentangledSSL’s frontier is significantly superior to that of JointOpt (in this case, the top-left corner is preferred), especially in the middle and left region. These two figures jointly demonstrate that DisentangledSSL more effectively disentangles shared and specific information than all existing methods.

While the experiments on synthetic datasets were conducted to provide a compelling proof of concept supporting our theoretical insights and claims, DisentangledSSL also outperforms existing baselines across real-world benchmarks such as MultiBench and molecule-phenotype retrieval tasks.

Table 1 shows the three versions of DisentangledSSL are unstable.

Please refer to the detailed response about the key observations from this table in the consolidated response to all reviewers above.

Dear reviewer nJHK,

As the discussion period is drawing to a close, we wanted to kindly request your feedback on our rebuttal. In our response, we have carefully tried to address all your concerns and also included additional experiments to further demonstrate the strengths of our work. We would greatly appreciate it if you could provide your feedback at your earliest convenience.

Thank you for your time.

Best regards,

Authors

We sincerely thank the reviewer for investing their time and, most importantly, for reconsidering our work based on the revised version. We truly appreciate their effort in helping us refine this draft. We are glad our revisions have addressed most of their concerns, especially the stability of our method.

Regarding the scale of performance improvement, the performance gains are, in general, limited due to the challenging nature of these tasks, as also shown in previous papers. Nevertheless, we see significant performance gain (>2%) for DisentangledSSL in multiple real-world tasks, for example, MUSTARD in MultiBench, and RXRX19a in molecule-phenotype retrieval (where the relative gain of retrieval accuracy is about 10%), as well as in the disentanglement measurement in Table 3.

In addition, we would like to emphasize the theoretical contribution of our paper. We address the challenging real-world setting where Minimum Necessary Information (MNI) is not attainable, with detailed and rigorous analysis in Sections 2.2 and 2.3. This case is underexplored in prior works and as we believe, is an extremely important contribution. While acknowledging the reviewer’s focus on key aspects of our work like empirical performance, we request and would appreciate a re-evaluation from both theoretical and empirical aspects.

Dear reviewer nJHK,

Since the deadline for reviewer response is today, we kindly request your feedback on our latest response about the significance of our performance gains on MultiBench, demonstrated in detail with references. If our response has addressed your concerns, we would greatly appreciate it if you could reconsider your score and share your feedback.

Thank you for your time and we truly appreciate your insights.

Best regards,

Authors

Dear reviewer nJHK,

Understanding that you may be busy, and with the author response deadline coming, we would like to take this last opportunity to summarize our discussions and the major updates and clarifications we made during the rebuttal to address your concerns.

1. The major remained concern of the reviewer is the ~2% performance gain in Table 1 (excluding MUSTARD). We have provided detailed explanations and references in our latest rebuttal to address this concern.
2. Other concerns the reviewer initially had, including the stability of model performance in Table 1, and the performance gain in Table 2, have been addressed during the rebuttal.
3. We thank the reviewer for engaging and for their care in discussing the empirical results. Since much of our contribution pertains to the theory/method, we would like to also emphasize these contributions of our paper which may not have been fully recognized earlier.

We thank the reviewer for their time and effort, and hope these clarifications address the concerns. If you find our rebuttal satisfactory, we hope that you would kindly consider re-evaluating our work.

Best,

Authors