Beyond DAGs: A Latent Partial Causal Model for Multimodal Learning

Thank you for your thoughtful and detailed feedback. We greatly appreciate your recognition of the potential impact of our approach, as well as your constructive suggestions for improving our work.

Q1: I thought there must be text to explain why this architecture is good, how to design algorithm to optimize ... But how is the objective in Section 3.2 related to your model in Section 3.1?...

R1: As mentioned in the introduction, assuming a DAG (Directed Acyclic Graph) structure can be a strong constraint compared to non-DAG models, as it requires an assumption about the causal direction. Additionally, identifying latent DAGs often necessitates various assumptions. In light of this, we pose the question: is a non-DAG assumption sufficient for certain applications? To address this, we propose a latent partial causal model depicted in Figure 1, which has the flexibility to encompass multiple possible DAG structures, as illustrated in Figure 2. Using this partial causal generative model, we examine a multimodal contrastive learning framework (inference model) to infer latent variables, given its success in various applications, such as CLIP. Alternative inference models, such as those maximizing likelihood, also exist. By applying certain assumptions to the proposed latent partial causal models, we prove that multimodal contrastive learning can identify latent coupled variables (e.g., relation between multimodal contrastive objective and the proposed model), as demonstrated in Theorem 4.1 and Corollary 4.2 on a hypersphere, and Theorem 4.3 and Corollary 4.4 on convex bodies. These theoretical results indicate that multimodal contrastive learning identifies latent coupled variables, and, if we assume these latent coupled variables are independent, multimodal contrastive learning can further identify them component-wise. Combined with linear ICA, this corresponds to achieving disentanglement.

Q2: Also, in Section 4, Theorem 4.1, I believe h is the composition of f_x and g_x, but why it only takes z_x (or z_t) as input? Where is m_x? And I did not follow the insights explained for Theorem 4.1. My concerns apply to Section 4.2.

R2: As we analysed in Theorem 4.1 and Theorem 4.3,the optimal normalized multimodal contrastive loss defined by Eq. (2) will effectively remove the influence of $\mathbf{m}_x$ in $\mathbf{h}$. Intuitively, for paired images and text, the contrastive loss functions to minimize the distance between their respective representations. In Figure 1, we can see that $\mathbf{m}_t$ and $\mathbf{m}_x$ are independent variables. If these components are not accounted for in the representations, the distance between the paired image and text representations will be influenced by $\mathbf{m}_t$ and $\mathbf{m}_x$. Since $\mathbf{m}_t$ and $\mathbf{m}_x$ are independent and can vary randomly, this introduces variability that makes it impossible to control the distance between the paired representations. This variability contradicts the goal of contrastive loss, which is to minimize the distance between matched image and text representations effectively. For more rigorous derivation, please refer to Eqs. 18 and 19.

Q3: Table 1 tries to evaluate the identificabiulity of the model, but how?

R3: Table 1 results is to verify our identifbaility analysis, e.g., on hyperspehre, latent coupled variables $\mathbf{z}_x$ are identified up to linear transformation, e.g., Theorem 4.1 and Corollary 4.2, on convex, latent coupled variables $\mathbf{z}_x$ are identified up to permutation , e.g., Theorem 4.3 and Corollary 4.4. Therefore, on hypersphere, we use R2, which quantifies the degree of any linear correlation between the estimated $\mathbf{\hat z}_x$ (representations obtained by multimodal contrastive loss) and the true $\mathbf{z}_x$, for eluviation. On the convex bodies, we use the MCC, a standard method for evaluating the recovery of independent components in nonlinear ICA, to eluviate that latent coupled variables $\mathbf{z}_x$ are identified up to permutation.

Q4: I think they are quite toy experiments, and I am not convinced about the advantage of the proposed method from the reported results

R4 This work does not propose a new method; instead, it provides an identifiability analysis for multimodal contrastive learning. As Theorem 4.1 and Corollary 4.2 demonstrate, multimodal contrastive learning has the ability to identify latent coupled variables up to a linear transformation. Thus, using linear ICA allows for component-wise recovery of latent coupled variables, which shows the potential for disentanglement in theory. To align with these theoretical results, we use pre-trained CLIP, followed by linear ICA, to verify our theoretical claims in epxeirments. In summary, we are not introducing a new method or verifying the advantages of the new method; rather, all our experiments are designed to align with and validate our theoretical claims.

Dear Reviewer PGzN,

Thank you for your reply and the opportunity for further discussion.

Before responding, we would like to provide some background to help the reviewer better understand our work.

One of our main contributions is explaining the success of multimodal contrastive learning. To achieve this, we propose investigating multimodal contrastive representation learning from a latent causal perspective. This approach allows us to examine high-level latent causal factors shared across modalities.

To this end, we first introduce a latent causal model assumption for the generative process of multimodal datasets. This provides a framework to interpret how such multimodal data are generated. Specifically, we propose our latent partial causal model, illustrated in Figure 1. Using this model, we analyze how the multimodal contrastive loss can be connected to the underlying latent causal model depicted in Figure 1.

Motivated by the insights in Section 3.2, we hypothesize that the representations learned via multimodal contrastive loss may identify latent coupled variables $z_x$ and $z_t$ within the proposed latent partial causal model. For instance, the learned representation $\hat z_x$ by multimodal contrastive loss could belong to the same equivalence class as the true latent variable $z_x$.

To rigorously establish this result, we introduce additional assumptions about the proposed latent partial causal model. For example, these include conditions such as Eq. (4) for hyperspheres or Eq. (7) for convex bodies. Under these assumptions, we prove that the features learned by the multimodal contrastive loss are indeed related to the true latent variables z_x and z_t. This is formalized in Theorem 4.1 and Corollary 4.2 for hyperspheres, which establish that $\hat z_x$= $A z_x + c$ , where A is a linear matrix, c is a constant. Similarly, Theorem 4.3 and Corollary 4.4 extend this result to convex bodies, showing that $\hat z_x$= $P z_x + c$, where P is a permutation matrix with scaling.

These results demonstrate that the features learned by multimodal contrastive loss is related with essential, generalizable patterns in the data, such as high-level latent coupled variables (e.g., $z_x$). This relationship provides theoretical support for the success of multimodal contrastive learning.

Furthermore, since we assume that $z_x$ has independent components, our identifiability analysis on hyperspheres (e.g., Theorem 4.1 and Corollary 4.2) demonstrates that for the learned representation $\hat z_x$, we have $\hat z_x = A z_x + c$. This result enables the application of linear ICA on $\hat z_x$ to recover the independent components of $z_x$ , coresponding to the potential for disentanglement.

We hope the above background helps the reviewer gain a deeper understanding of our work.

For your concerns:

1. As mentioned in Introduction (e.g., lines 061–062), 'Given the proposed.., we analyze it within the widely used multimodal contrastive learning paradigm (e.g., inference).' Specifically, we analyse the proposed model using the multimodal contrastive loss defined in Eq. (1), which belongs to a broader class of multimodal contrastive losses, including multimodal contrastive loss on hypersphere.

2. Indeed, h_x is a composition of f_x and g_x. Without any further analysis, it would naturally include both z_x and m_x. However, by considering the optimal solution of the multimodal contrastive loss and the independence properties in the proposed latent partial causal model, e.g., m_x is independent z_x, it can be shown that h_x do not depend on m_x. As we metioned in R2, for a more rigorous derivation, please refer to Eqs. 18 and 19, which demonstrate that the partial derivative of h_x with respect to m_x is equal to 0, implying that h_x does not depend on m_x. (similarly for h_t). Additionally, we provide an intuitive explanation in R2.

Thank you for your further concern.

Before responding, we re-reviewed our process to better understand your concerns. During this review, we noted your previous comment: "because because hx(mx, zx) = ht(mt, zt) does not necessarily imply \nabla_mx hx(mx, zx) = \nabla_mx ht(mx, zx)". We are unsure whether this is a typo. If "\nabla_mx ht(mx, zx)" is not a typo, please note that ht corresponds to the encoder for text data, and thus the effective input should be (mt, zt), i.e., "\nabla_mx ht(mx, zx)"should be "\nabla_mx ht(mt, zt) ". If it is a typo, please refer to the below.

Consider the paired data $((m_x, z_x), (m_t, z_t))$, the complete expression of $h_x(m_x, z_x)=h_t(m_t, z_t)$ should be $h_x((m_x, z_x), (m_t, z_t))=h_t((m_x, z_x), (m_t, z_t))$. This is because during the training process in multimodal contrastive learning, two different encoders are applied to two distinct modalities. As a result, in $h_x((m_x, z_x), (m_t, z_t))$, $(m_t, z_t)$ term is masked. Similarly for $h_t((m_x, z_x), (m_t, z_t))$. We now proceed to prove that their derivatives are equal everywhere.

The derivative of $\frac {\partial {h} _ x( {m} _ x,{z} _ x, m_t, z_t )} {\partial {m}_x}$ is defined as:

$\frac {\partial {h} _ x({m} _ x,{z} _ x, m_t, z_t )} {\partial {m}_x} = \lim _ {e \rightarrow 0} \frac {{h} _ x({m} _ x + e,{z} _ x, m_t, z_t)- {h} _ x({m} _ x,{z} _ x, m_t, z_t) } {e}$ .

Similarly, the derivative of $\frac {\partial {h} _ t( {m} _ x,{z} _ x, {m} _ t,{z} _ t)} {\partial {m}_x}$ is defined as:

$\frac {\partial {h} _ t({m} _ x,{z} _ x, {m} _ t,{z} _ t)} {\partial {m}_x} = \lim _ {e \rightarrow 0} \frac {{h} _ t( {m} _ x + e,{z} _ x, {m} _ t,{z} _ t)- {h} _ t({m} _ x,{z} _ x, {m} _ t,{z} _ t) } {e}$ .

Since $h_x(m_x, z_x, m_t, z_t)=h_t(m_x, z_x, m_t, z_t)$, and considering that we define paired data based on z as $p(z_t|z_x)$ in Eq. (4) (or (7)), the pairing $({m} _ x + e,{z} _ x, m_t, z_t)$ holds despite the disturbance e on m_x. Consequently, ${h} _ x({m} _ x + e,{z} _ x, m_t, z_t) = {h} _ t({m} _ x + e,{z} _ x, {m} _ t,{z} _ t)$.

Given the above, $\frac {\partial {h} _ x({m} _ x,{z} _ x, m_t, z_t )} {\partial {m}_x} = \lim _ {e \rightarrow 0} \frac {{h} _ x({m} _ x + e,{z} _ x, m_t, z_t)- {h} _ x({m} _ x,{z} _ x, m_t, z_t) } {e} = \lim _ {e \rightarrow 0} \frac {{h} _ t({m} _ x + e,{z} _ x, m_t, z_t)- {h} _ t({m} _ x,{z} _ x, m_t, z_t) } {e} = \frac {\partial {h} _ t({m} _ t,{z} _ t, {m} _ x,{z} _ x )} {\partial {m}_x}$ (Here $m _ x \perp m _ t, z _ t$, so that $\frac {\partial {h} _ t({m} _ t,{z} _ t, {m} _ x,{z} _ x )} {\partial {m}_x}=\frac {\partial {h} _ t({m} _ t,{z} _ t )} {\partial {m}_x}=0$). Therefore, the partial derivatives are equal.

For the reference, please see Eq. (16), which indicates that if the function h, analogous to h in our work, depends on $m_1$ or $m_2$ (corresponding to our m_x and m_t), then $|h_1-h_2|>0$, h_1 and h_2 are analogous to h_x and h_t in our work. In other words, h cannot depend on modality-specific information such as $m_1$ or $m_2$, to satisfy $h_1= h_2$.

Thank you sincerely for dedicating your time to reviewing our work and, most importantly, for meticulously checking our proof step by step.

We would like to highlight two key contributions of our work (More detailed contributions are summarized in the Introduction section): 1) Unlike traditional approaches relying on DAGs, we demonstrate that latent causal models with non-DAG assumptions among latent causal variables are sufficient for certain applications, such as explaining the success of multimodal contrastive learning. 2) To the best of our knowledge, this is the first work to provide theoretical guarantees for the potential of disentanglement in models trained via multimodal contrastive learning, thereby pushing the boundaries of pre-trained models like CLIP.

The condition of exact modality matching in our theoretical results is a commonly used assumption in previous works analyzing multimodal data, e.g., [1, 2]. Relaxing this condition represents a non-trivial and compelling direction for future research.

[1] Daunhawer, Imant, et al. "Identifiability results for multimodal contrastive learning." ICLR (2023).

[2] Yao, Dingling, et al. "Multi-view causal representation learning with partial observability." ICLR (2004).

Thank you for your thoughtful and detailed feedback. We appreciate your recognition of the potential impact of our approach and your constructive suggestions for improving our work.

Q1: The paper is strongly based on the results presented in [1] and provides similar insights as previous works [2,3],

R1: One of our primary motivations is to explore whether partial causal models are sufficient for specific applications. To address this, we introduce latent partial causal models and use multimodal contrastive learning as an illustrative example. We demonstrate that: (1) partial causal models can theoretically support the success of multimodal contrastive learning by enabling the identification of latent coupled variables, and (2) these models can also facilitate the disentanglement potential of multimodal contrastive learning. Our findings suggest that partial causal models may be adequate for certain applications, offering a novel insight: full DAG assumptions in latent causal analysis might be unnecessary for some practical cases. This avoids the restrictive and often impractical assumptions associated with full DAGs, an aspect previously overlooked. Additionally, while some results extend single-modal analysis from [3], our contributions include: (1) the development of Theorems 4.1 and 4.2, bridging the theoretical gap between single-modal and multimodal analysis, showing the applicability of certain results from [3] in a multimodal context, as elaborated in subsequent paragraphs, and (2) highlighting the underexplored disentanglement potential of pre-trained multimodal models like CLIP. This potential, achievable through methods as straightforward as linear ICA, warrants consideration for future applications

Q2: Could the authors better clarify the centrality of the non-DAG assumption and why it is necessary for the insights in this work?

R2: The main motivation of this work is that, in general, causal analysis requires satisfying DAG (Directed Acyclic Graph) assumptions. For example, when modeling multimodal data, there are three possible DAG structures, as shown in Figure 2. Each DAG requires specific assumptions, such as defining the causal relationship between zx and zt. Additionally, identifying these latent causal variables and the DAG structure typically demands further assumptions, such as specific parametric constraints on the variables or limitations on the function class describing the causal relationships. However, verifying these assumptions (including graph structure and parametric or function class constraints) is often difficult in real-world applications.

Motivated by this challenge, a natural question arises: are DAG constraints always necessary? Could non-DAG causal models be sufficient for certain applications? To address this, we propose the latent partial causal model shown in Figure 1, in which a undirected edge between $\mathbf{z}_x$ and $\mathbf{z}_t$ is provided so that we avoid further assumptions on the causal direction between $\mathbf{z}_x$ and $\mathbf{z}_t$. By combining this partial model with multimodal contrastive learning as an example, we demonstrate that multimodal contrastive learning can identify latent coupled variables in the proposed latent partial causal model, $\mathbf{z}_x$ and $\mathbf{z}_t$, supporting the effectiveness of multimodal contrastive learning and revealing its theoretical potential for disentanglement.

These findings indicate that, for certain applications, non-DAG assumptions can be sufficient, removing the need for stringent DAG constraints. This insight is novel and significant, as it shows that the strict requirements of traditional DAG-based causal analysis may not always be necessary, thus broadening the scope of practical causal modeling.

We thank Reviewer B7j4 for recognizing its identifiability analysis, disentanglement implication, a novel perspective on the latent partial causal model setting. We address the reviewer’s questions below.

Q1: Section 3.2 about prior matching and information preservation shares a lot in common with many concepts being explored in [1], and the mutual information perspective in [2]... The paper lacks discussions/comparisons with these perspectives.

R1: Indeed, the alignment-uniformity perspective and mutual information (information preservation) have been discussed in prior work within the context of single-modal learning. However, these two aspects are typically investigated separately. In this work, we extend these perspectives to a multi-modal setting and, more importantly, combine them to explore their role in solving inverse problems. This combined way provides a new insight: multimodal contrastive learning has the potential to address inverse problems. Thank you for the suggestion. We have updated the paragraph on line 284 to include a clear comparison and emphasize our novel insight into the relationship between contrastive learning and solving inverse problems.

Q2: The condition in Theorem 4.1, looks strong. It could be hard to achieve in real-world data.

R2: We agree that this condition stems from theoretical analysis, which does not take optimization challenges into account. Achieving such conditions in real-world applications can be difficult, particularly due to issues like local optimization or limited real dataset. However, our primary theoretical focus is on identifiability results, where it is generally assumed that global optimization can be achieved. Further theoretical analysis related to optimization is indeed challenging and beyond the scope of this paper.

Q3: Compared to unimodal representations from SSL methods and the analysis in [3], it is unclear what specific gain can be attained from the multimodal representations.

R3: Nice question. One of our primary motivations is to explore whether partial causal models are sufficient for specific applications. To address this, we introduce latent partial causal models and use multimodal contrastive learning as an illustrative example. We demonstrate that: (1) partial causal models can theoretically support the success of multimodal contrastive learning by enabling the identification of latent coupled variables, and (2) these models can also facilitate the disentanglement potential of multimodal contrastive learning. Our findings suggest that partial causal models may be adequate for certain applications, offering a novel insight: full DAG assumptions in latent causal analysis might be unnecessary for some practical cases. This avoids the restrictive and often impractical assumptions associated with full DAGs, an aspect previously overlooked. Additionally, while some results extend single-modal analysis from [3], our contributions include: (1) the development of Theorems 4.1 and 4.3, bridging the theoretical gap between single-modal and multimodal analysis, showing the applicability of certain results from [3] in a multimodal context, as elaborated in subsequent paragraphs, and (2) highlighting the underexplored disentanglement potential of pre-trained multimodal models like CLIP. This potential, achievable through methods as straightforward as linear ICA, warrants consideration for future applications.

Q4 Can the authors give some intuition on how the modality-specific variables are got rid of in Theorem 4.1 .....

R4 This is also a good question. Let's consider the generative process illustrated in Figure 1. Intuitively, for paired images and text, the contrastive loss functions to minimize the distance between their respective representations. In Figure 1, we can see that $\mathbf{m}_t$ and $\mathbf{m}_x$ are independent variables. If these components are not accounted for in the representations, the distance between the paired image and text representations will be influenced by $\mathbf{m}_t$ and $\mathbf{m}_x$. Since $\mathbf{m}_t$ and $\mathbf{m}_x$ are independent and can vary randomly, this introduces variability that makes it impossible to control the distance between the paired representations. This variability contradicts the goal of contrastive loss, which is to minimize the distance between matched image and text representations effectively. For the case of where $\mathbf{m}_t$ and $\mathbf{m}_x$ dominate, we believe that the theoretical results should still hold. However, there may be significant challenges in optimization during implementation. For instance, the features learned by multimodal contrastive loss may incorporate both $\mathbf{m}_t$ and $\mathbf{m}_x$ information to some extent. The degree to which this information is included largely depends on the optimization process.

Thank the authors for providing detailed clarifications, and I would like to maintain my positive score.

We thank Reviewer KBLq for taking the time to review our submission and for recognizing its well-organized structure, clear writing, strong substantiation, and contributions to an interesting and important problem, as well as the new insights it provides. We address the reviewer’s questions and critiques below.

Q1: There are fundamental issues for possible DAGs in Figure 2... There exists a real world that we measure with various sensors, such as the camera (generating images from various views) or textual descriptions of the scene... Figure 2(b) and Figure 2(c) can also be captured as inference on the first DAG i.e. Figure 2(a).

R1: We agree that the scenario described by the reviewer can be effectively modeled using a shared latent variable across modalities. However, there are real-world examples, such as text-to-image generation (e.g., in datasets like MNIST) and image captioning as seen in datasets like CelebA-Dialog, where the structures depicted in Figures 2(b) and 2(c) are more appropriate. We acknowledge that for many correlation-based inference tasks, using slightly mis-specified models can yield results comparable to those obtained with the true model. For instance, employing the model in Figure 2(a) for both learning and inference on data generated by the true data generation processes depicted in Figures 2(b) and 2(c) may produce results similar to those obtained with the corresponding true model. However, the primary advantage of learning a model that aligns with the true process extends beyond correlation-based inference tasks. A true causal model enables intervention and counterfactual analysis, which are not possible with a misspecified model.

Q2: Theorem 3.1 is interesting and provides insights into the CLIP objective ..... I am unsure if it adds a lot of value.

R2: Theorem 3.1 offers an interesting insight into how multimodal contrastive loss is related to addressing inverse problems, as analyzed following the theorem. It is important to note that this insight was not provided in Wang and Isola (2020). Furthermore, Theorem 3.1 serves as a cornerstone for developing Theorems 4.1 and 4.3. In other words, it acts as a bridge connecting the multimodal contrastive loss in Eq. (1) to the results in Theorems 4.1 and 4.3.