Towards the Causal Complete Cause of Multi-Modal Representation Learning

We sincerely appreciate Reviewer eaJB's constructive feedback and the time and effort dedicated to the review process. We are also grateful for the recognition of our work and sincerely hope the following responses can eliminate the concerns.

Response to W1

We appreciate the suggestions and apologize for any misunderstandings. To distinguish it from traditional identifiability, we change the word to "Causal Identifiability." According to Pearl (2009) and Lee & Bareinboim (2020), causal identifiability refers to establishing the effect of intervening on a set of variables on another set, using observational or interventional distributions under given assumptions. It permits the inclusion of interventional terms if they are uniquely determined from the observational distribution (Shpitser, 2012). Based on these, Theorem 3.2 is derived. When the model satisfies local invertibility, we can uniquely recover the variable distribution from the observed data, thereby ensuring causal identifiability and estimation of $C^3$.

Response to W2

The experiments on computational cost in Appendix H.4 (Figure 7) indicate that introducing $C^3$R increases the computational cost by less than 1.3x compared to the original.

Response to W3

We provide an outline to explain how it "handles hidden confounding":

• Theoretically: The "hidden confounding" issues correspond to the relaxation of the exogeneity assumption, which requires satisfying causal identifiability of $C^3$ in the environment that contains hidden confoundings. To achieve the relaxation, we introduce an instrumental variable $V$ to eliminate confounding effects, thus achieving causal identifiability regardless of whether hidden confounding exists (Section 3.2). We model $V$ using the improved self-attention mechanism, which employs alignment scores to capture causal factors $F_c$, theoretically constrain $Z$ to only contain $F_c$, and eliminate confounding effects (L220-257 and Appendix A.3).
• Methodology: Based on theoretical results, we constrain the causal completeness of the representations with $C^3$ risk. The concept of causal completeness also means without "hidden confounding" in causality (Pearl, 2009). We propose $C^3$R to constrain it by reducing $C^3$ risk. It is achieved through a twin network: the real-world branch uses $V$ to adjust $Z$ so that it only contains $F_c$ for accurate prediction, achieving causal sufficiency and eliminating hidden confounding; the hypothetical branch employs gradient-based counterfactual modeling to further calibrate the causal factors, constraining causal necessity and further filtering out hidden confounding (L258-307 and Appendix D.4). Table 1-8 prove its effectiveness.

Response to Minor 1

We sincerely appreciate the suggestions and would like to provide an outline for illustration (further emphasized in Section 3.1):

• Meaning of Causal Sufficiency: Our concept of causal sufficiency (Definition 3.1) is adapted from Definition 9.2.2 in Pearl (2009), i.e., “setting x would produce y in a situation where x and y are in fact absent” which is considered the original definition of causal sufficiency. It also aligns with another type of illustration in causality literature, e.g., “the absence of latent confounder”, as we utilize instrumental variable and twin network in Sections 3.3 and 4 to eliminate confounders, aiming to satisfy causal sufficiency.
• Clarification on Assumptions: Previous works rely on an exogeneity assumption, assuming there are no confounders in environments. In contrast, we relax these assumptions to account for possible confounders in practice (L80–99 and Appendix D.3). It does not alter the meaning of causal sufficiency; it means that we do not assume that there is no confounding in the environment, but introduce instrumental variables to eliminate the impact of confounding to relax the assumption.

Response to Minor 2

We sincerely appreciate the suggestions and have carefully reviewed the mentioned paper. Below is a brief illustration of the differences (cited and supplemented in Section 7). Wang & Jordan aim to construct measures of non-spuriousness and disentanglement. Their exploitation of causal necessity and sufficiency concept is to align it with non-spuriousness and “invoke” the corresponding measure in Pearl (2009) to construct the measure of non-spuriousness. We focus on modeling the concept of causal sufficiency and necessity itself in MML. We relax the exogeneity and monotonicity assumptions that previous works depend on, including Wang & Jordan, and propose a new method to measure and constrain MML-specific causal completeness. Thus, although both works draw inspiration from Pearl (2009), they differ significantly in problem, theoretical framework, methodology, and experimental validation.

Regarding Comments

We appreciate the reviewer's suggestions and have polished the text accordingly, i.e., changing “with both” to “be both”, “conduct” to “construct”, and “a” to “an”.

We sincerely appreciate the reviewer fkSa's constructive feedback and the time and effort dedicated to the review. We are grateful for the recognition of our work and sincerely hope the following responses can eliminate the concerns.

Response to W1

We sincerely appreciate the suggestions and have refined Fig.1 accordingly. The examples in Fig.1 are based on specific tasks and data conditions (L129-155). A sufficient and necessary feature can be "flat duck bill" as its presence indicates "duck" and every "duck" sample includes it. We also added the textual modality, e.g., "A duck has a flat bill and orange webbed feet standing with its wings folded" as suggested for better clarification.

Response to W2

We agree “in the absence...” but would like to kindly clarify that the SCMs in Fig.2 are conducted under different settings, aiming to illustrate potential confounding issues in MML instead of true data-generating direction (L129-155). Left is for how MML sample $X$ is generated based on causal generating mechanism; Right is to align with the practical MML process, where factors are typically coupled for predicting $Y$. We apologize for any ambiguity that may caused by the caption and have refined it accordingly.

Response to W3

We sincerely appreciate the suggestions and would like to clarify that our concept of causal sufficiency (Definition 3.1) is adapted from Definition 9.2.2 in Pearl (2009), i.e., “setting x would produce y where x and y are in fact absent”. It aligns with the mentioned “the absence of latent confounder”, as we utilize instrumental variable and twin network to eliminate confounders (Sections 3.3 and 4), aiming to satisfy causal sufficiency. We further emphasized it in Section 3.1 according to the valuable suggestion.

Response to W4, Theoretical Claims 1&2

• We have illustrated “local invertibility” in Proposition D.3 and will further elaborate on it immediately following Theorem 3.2 as suggested. Briefly, it states that the model can uniquely recover the distribution of exogenous variable $s$ from the conditional distribution of $Y$ given its parents $Pa(Y)$.
• We have provided a brief explanation about non-exogeneity, e.g., $P(Y_{do(Z=c)}) \neq P(Y \mid Z=c)$ in L185-191 with detailed in Appendix D.3. We will further refine this immediately following the theorem.
• Compared to Theorem 3.2, the non-monotonicity in Theorem 3.3 allows the effect of $Z$ on $Y$ to vary in direction or intensity. The key modification is introducing $V$ for piecewise estimation through integration (Eq.5).

Response to W5

We sincerely appreciate the suggestions and provide a brief derivation following Theorem 9.2.15 in (Pearl, 2009): $C^3(Z)$ can be decomposed into (i) when $Z\neq c$ occurs with probability $1-P(Z=c)$, its contribution reflects $C^3_{su}(Z)$ for $Z=c$; (ii) when $Z=c$ occurs with $P(Z=c)$, its contribution reflects $C^3_{ne}(Z)$ of $Z=\bar{c}$ on $Y$. By normalizing, we get $C^3_{su}(Z) = \frac{C^3(Z)}{1-P(Z=c)}$ and $C^3_{ne}(Z) = \frac{C^3(Z)}{P(Z=c)}$ as Eq.3 and Eq.4. We will add it in Appendix A.1 for better understanding.

Response to W6 & Theoretical Claim 3

The intuition is twofold:

• Aligning with the modeling objectives of $V$ (L220–248): The self-attention mechanism, dynamically measures the alignment score between modalities, can emphasize the important features for $Y$ across modalities (i.e., achieves higher scores on $F_c$) while downplaying $F_s$. Intuitively, using self-attention-based $V$ on $Z$ with distance penalties satisfies the goal that "constrain $Z$ only contains $F_c$".
• Higher efficiency: Although custom-designed networks can also achieve the above goal, the self-attention mechanism is typically more lightweight without substantially increasing model complexity.

Response to Q1

The learned representations can "be mapped to interpretable latent concepts". Such a representation must both fully explain the decision (sufficiency) and be indispensable (necessity), revealing the most important factors behind predictions. For example, a causally complete representation captures all critical factors for classifying "happy", e.g., visual (upturned mouth corners, wide-open eyes) and audio cues (high pitch) where removing "upturned mouth corners" may result in "angry".

Response to Q2

The results in Table 2 (missing modality) and Table 8 (data noise) show that $C^3$R achieves stable performance improvements in domains with real-world data complexities.

Regarding Essential References Not Discussed

We appreciate the suggestion and carefully reviewed the papers (cited and supplemented in Section 7). Schölkopf et al. present a review for causal inference; the rest four mainly focus on identifiability under different settings and problems, e.g., weak supervision and partial observability. We focus on the causal sufficiency and necessity concept, aiming to explore and model it for MML, where the problem, theory, method, and experiments are all different.

We sincerely appreciate Reviewer ZspQ's feedback and the time and effort dedicated to the review. We are also grateful for the recognition of our work and sincerely hope the following responses can eliminate the concerns.

Response to W1 & Q3

• The true counterfactual effect involves unobserved outcomes, making direct modeling difficult (Pearl, 2009). Existing work typically adopts the "minimal change" principle to estimate counterfactuals, which is proven to align with true counterfactual effects (Galles & Pearl, 1998; Kusner et al., 2017). For instance, Chapters III–V in Wachter et al. (2017) demonstrate that by making only minor adjustments to the treatment variable while preserving the distribution of other covariates, the counterfactual samples maintain the original data’s characteristics, ensuring accurate counterfactual effect estimates. Based on these results, we leverage gradient intervention to satisfy the "minimal change" principle for counterfactual effect estimation with theoretical guarantees (Appendix D.4). To assess whether the generated counterfactual data adhere to the principle for accurate estimation, we develop a distribution consistency test with Wasserstein distance $D_w$, i.e., whether the distribution of the covariates matches that of the original data. The lower $D_w$ shown below proves that our method satisfies the principle.
• According to the suggestions, we conduct a toy experiment on LCKD and NYU Depth V2 to demonstrate credibility. We follow (Galles & Pearl, 1998) to make manually curated examples and select the recently proposed transport-based counterfactual modeling method (Lara et al., 2024) as another baseline. The table below shows the advantages of our method, i.e., superior accuracy and lowest computational cost. |Method|Acc|Calculation overhead|$D_w$| |-|-|-|-| |$C^3$R|77.6|$1\times$|0.9| |manually curated|77.4|$4.9\times$|0.7| |Transport-based|75.2|$2.3\times$|2.6|

Response to Q1

We have conducted experiments to validate "$C^3$ risk serves as an accurate indicator":

• Appendix H.5: We calculate the heatmaps of samples using $C^3$ risk in Fig.7, where high-weight elements correspond to low $C^3$ risk. If low $C^3$ risk reflects causally sufficient and necessary factors, then reducing the weights of these elements would degrade model performance. Table 7 shows that reducing the weights of low $C^3$ risk elements indeed lowers performance, confirming that $C^3$ risk is an accurate indicator.
• Section 6.2: We conduct experiments on MMLSynData, which includes four types of generated data, i.e., sufficient and necessary causes (SNC), sufficient but unnecessary causes (SC), necessary but insufficient causes (NC), and spurious correlations (SP). By evaluating the correlation between the representation learned by minimizing $C^3$ risk and SNC, we can validate whether low $C^3$ risk refers to causal sufficient and necessary causes. Figures 3 & 6 indicate that $C^3$R significantly increases the correlation between representations and SNC, proving the accuracy.
• Section 6.2 & Appendix H: If $C^3$ risk can be the indicator, then $C^3$R should lead to performance improvements by minimizing $C^3$ risk. Tables 1-8 show that $C^3$R achieves significant performance gains across all baselines, proving effectiveness.

Response to Q2 & Comments

We have conducted experiments on datasets with "high spurious correlations" and provide the outline below:

• Quantitative (Section 6.2, Appendices D.5 and H.3): We conduct experiments on MMLSynData, which contains SNC, SC, NC, and SP as mentioned in "Response to Q1 (2)". Different degrees of $D_{sp}$ are set to control the level of spurious correlations. By evaluating the correlation between the learned representation and SP/SNC, we can validate the effectiveness of the corresponding model in mitigating spurious correlations/learning causal complete representation. Figures 3 and 6 show that even with high spurious correlation ($D_{sp}=0.6$), $C^3$R markedly reduces the correlation with SP (0.4 -> 0.1) while significantly increasing the correlation with SNC (0.5 -> 0.9), proving advantages.
• Qualitative (Appendix H.5): As in "Response to Q1", the visualization shows that $C^3$R learns causally complete representations and effectively eliminates spurious correlations.

Regarding Essential References Not Discussed

We sincerely thank the suggestion and carefully reviewed the paper, finding differences with ours in problem, theory, method, experiments, etc. (cited and added in Section 7). Kim et al. aim to address modality bias in multispectral pedestrian detection tasks, e.g., "prediction on ROTX without thermal features". They conduct SCMs for analysis and propose CMM for the tasks. Differently, we focus on the concepts of causal sufficiency and necessity and are the first to explore them in MML. We propose a new theoretical framework and a plug-and-play method to learn causal complete representations, which can also be embedded in CMM.