Plug-and-play Feature Causality Decomposition for Multimodal Representation Learning

We are truly grateful for your insightful suggestions and are pleased to provide our responses.

---

Weaknesses

W1. Backdoor-adjustment has been widely used (novelty is limited).

A1. Backdoor-adjustment has been widely used in various kinds of tasks, including graph representation learning, multimodal sentiment analysis [1] etc. The backdoor-adjustment itself has been proposed for a long time [2], which usually performed as a tool for debiasing. Many researchers have dedicated to employ it for removing the causal effect of confounders in their systems, such as generating counterfactual samples, working together with frontdoor-adjustment etc. So do we. However, different with them:

• (1) We utilize it in our theoretical derivation process for removing the causal effect of unimodal uncertainty noise on the task prediction.

• (2) We just utilize it in the theoretical level to avoid the potential irrationality and poor diversity of counterfactual samples. For example, in [3], the backdoor-adjustment is employed to generate counterfactual samples via replacing the background of samples within a batch. The diversity of generated samples are highly related to the dataset scale, where only backgrounds contained in the dataset can be used for intervention. Different with them, we apply backdoor-adjustment on the theoretical derivation process without explicitly generating counterfactual samples, which can avoid the potential impact of dataset scale (the diversity of confounder).

• (3) FCD is a plug-and-play module that can be implemented to the existing methods for unimodal uncertainty noise removal.

In short, FCD is a different representation learning plug-and-play module compared with existing ones. Just like the Bayes’ theorem in probabilistic theory [4], the backdoor-adjustment is a basic conception in causality theory.

[1] Changqin Huang et.al. AtCAF: Attention-based causality-aware fusion network for multimodal sentiment analysis. Information Fusion 2025.

[2] Judea Pearl. Causality. Cambridge university press 2009.

[3] Sarah Rastegar et.al. Background no more: Action recognition across domains by causal interventions. Computer Vision and Image Understanding 2024.

[4] Jaynes, E.T.: Probability theory: The logic of science. Cambridge university press 2003.

---

W2. The actual role of causality remains largely conceptual.

A2. We aim to extract the complementary and consistent information whilst removing the uncertainty noise mixed in the unimodal features. As it has been widely proved effective for defounding (debiasing), backdoor-adjustment and causality are also employed in our work. However, we aim to provide a generic plug-and-play module. To this end, we hope to simplify the actual complexity of FCD as much as possible through theoretical analysis and deduction. Therefore, we propose Theorem 4.1, which proves that the unimodal uncertainty noise can be removed by Eq. 6 based on backdoor-adjustment under certain conditions. Although it seems to be “conceptual”, the actual roles of causality and backdoor-adjustment are vital in our paper.

---

W3. Validation of causal structure.

A3. Please refer to the A1 in Question below.

---

W4. Experiments on larger-scale dataset.

A4. Please refer to A2 in Question below.

---

Questions

Q1. The motivation of SCM construction.

A1. Previous researches mainly focused on the better utilization of the complementary and consistent information (such as fusion strategy, dynamic fusion, etc.), or mitigating aleatoric uncertainty within each modality. We summarize the previous related works and empirically abstract the unimodal feature into 3 types of factor (synergistic, unique and redundant) corresponding to work [1], which proposed these concepts and validated them from both theory and demo experiments.

We acknowledge your suggestion that data-driven causal structure validation helps to improve the persuasiveness of the proposed SCM, though we note that this direction has rarely been explored in the existing literature [2-5]. To validate the causal structure of our proposed SCM and the correspondence with [1], following [1], we conduct the experiments using Self-MM on CMU-MOSI testing set, where the normalized information increment is used as metric. Please refer to [1], for their theoretical derivation and explanation.

We varify the causal relationship between $\mathbf{z}$, $\mathbf{h}$, $\mathbf{s}$, $\mathbf{u}$, $\mathbf{r}$, $\hat{y}$. We first store variables of each modality, and then apply PCA on the hidden embeddings ($\mathbf{z}$, $\mathbf{h}$, $\mathbf{s}$, $\mathbf{u}$, $\mathbf{r}$) along the dimension axis. Therefore, all vectors are reduced in dimension to scalars. Then, we apply Causal Additive Models algorithm for causal discovery. We repeat this procedure for 1000 times to calculate edge confidence. Since figures are not allowed in rebuttal period, we simply describe the result as follow:

Each element of the table stands for the frequency of edge occurrence in repeated experiments. The higher the frequency, the stronger the causal relationship between two variables. It is apparently that synergistic $\mathbf{s}$ and modality-specific task-related $\mathbf{u}$ have stronger causal relationship with the target $\hat{y}$. And the overall results indicate the causal effectiveness of our proposed SCM structure.

[1] Álvaro Martínez-Sánchez et.al. Decomposing causality into its synergistic, unique, and redundant components. Nature Communications 2024.

[2] Yongduo Sui et.al. Causal attention for interpretable and generalizable graph classification. SIGKDD 2022.

[3] Changqin Huang et.al. AtCAF: Attention-based causality-aware fusion network for multimodal sentiment analysis. Information Fusion 2025.

[4] Yanan Zhang et.al. Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspective. NeurIPS 2024.

[5] Weixing Chen et.al. Cross-Modal Causal Intervention for Medical Report Generation. TIP 2025.

[6] Peter Bühlmann et.al. CAM: Causal additive models, high-dimensional order search and penalized regression. The Annals of Statistics 2014.

---

Q2. Empirical evaluations on larger-scale datasets (e.g., MIMIC-CXR).

A2. We evaluated our method on different datasets with different scales (CMU-MOSI with $2199$ samples, UPMC Food101 with nearly $10^5$ samples, please refer to Appendix. C for more details.). Besides, as we mentioned in Weaknesses A1, since FCD doesn’t acutally generating counterfactual samples, the impact of dataset size is theoretically reduced.

Although we tried to apply for MIMIC-CXR dataset as son as we can, it requires certain Certification (the CITI program) and verification to access the dataset and it is too large to donwload and train, which may delay the rebuttal process. Practically, following CMCRL [1], we use the preprocessed dataset, and conduct the quantitative experiment on CMCRL. Because of the complexity of CMCRL and the scale of dataset, the experiment is still ongoing. We will update the results as soon as they come out.

[1] Weixing Chen et.al. Cross-Modal Causal Intervention for Medical Report Generation. TIP 2025.

We truly value your expertise and the careful consideration you've given to our work, which has helped us refine our approach.

---

Cons (Weaknesses)

W1. Citation of Theorem 4.1.

A1. We are sorry for this careless. Theorem 4.1 is based on the casual inference [1], information theory [2], and probabilistic theory [3]. We will formally add the necessary citation in the camera-ready version if accepted.

[1] Judea Pearl. Causality. Cambridge university press 2009.

[2] Ronghao Lin et.al. Multi-Task Momentum Distillation for Multimodal Sentiment Analysis. Affective Computing 2024.

[3] E. T. Jaynes. Probability theory: The logic of science. Cambridge university press 2003.

---

W2. Typos in the paper.

A2. Thanks again for your careful review. We will revise these typos and recheck the content of the paper.

---

W3. Different datasets in ablation study and case study.

A3. The reason why we conducted these experiments on different datasets is mainly because of the characteristics of the datasets themself. We didn’t conduct case study on CMU-MOSI dataset because the visual modality data are videos, which can not be directly displayed in the paper. Thus, we perform the ablation study on the MVSA-S using CLMLF and the results are as follows.

From the table, we can find that it has the similar tendency with the ablation study on CMU-MOSI using Self-MM. Eq. 16 and Eq. 13 are the ablation cases for task-related features fusion and shortcut in SDA module. The results also show the collective effect of our designs, which is consistent with Section 5.4.

---

Questions

Q1. Introduce FCD to existing works and the motivation behind tuning the hyperparameter.

A1. As you mentioned, previous works have their unique designs. So we abstract them into a unified process framework (Fig. 2 (a)), which mainly consists three parts (unimoda encoder, fusion module and prediction head). FCD is designed to decompose the synergistic, unique and reduntant features from unimodal representations, so it is attached between the unimodal encoders and the fusion module (Fig. 2 (b)).

Since FCD is a plug-and-play module, we only tune the hyperparameters involved by FCD and fixed the original ones. We report the hyperparameter search process in Appendix E. We first ranging each hyperparameter in {$0.5, 0.05, 0.005, 0.0005, 0.00005$} to find the suitable scale, then a more fine-grained search is performed in this suitable scale to find the final results. Please refer to Appendix E for more details about the tuning process.

---

Q2. The impact of the number of modality on FCD.

A2. The number of modelity would significantly affect the training time cost because of the pairwise comparison $\mathcal{L}\_{\text{SDA}}$, which has $O(M^2)$ complexity. The purpose of designing this $\mathcal{L}\_{\text{SDA}}$ is to reduce the differences between different modal distributions, ensuring that the extracted synergistic features are mapped into the same representation space and indeed shared among $M$ modalities. Moreover, the calculation of Sinkhorn divergence may also become time-consuming as $M$ increases. To improve it, we can randomly sample two modalities in each iteration. Then only one Sinkhorn divergence between two distributions is calculated to achieve approximate alignment effect.

---

Q3. The difference between FCD and methods following the disentanglement paradigm. like DMD[1] and its follow-up papers.

[1] Decoupled Multimodal Distilling for Emotion Recognition, CVPR 2023.

A3. FCD not only focus on the disentanglement of modality-invariant and modality-specific information (as DMD[1] did), but also explicitly considers the unimodal uncertainty noise contained in each modality caused by data acquisition processes and sensor characteristics. In short, FCD takes the modality-invariant (consistent information), modality-specific task-related (complementary information) and modality-specific task-agnostic (unimodal uncertainty noise) into consideration simultaneously. Then, based on causal inference, we propose to remove the unimodal noise mixed in the modality-specific feature to obtain the complementary information. Meanwhile, the consistent information is also extracted. After obtaining both consistent and complementary information, the task-related information can be better utilized. Moreover, FCD is a plug-and-play module that can be integrated to existing methods to improve their performances by eliminating the unimoda uncertainty noise.

[1] Yong Li et.al. Decoupled Multimodal Distilling for Emotion Recognition, CVPR 2023.

We greatly appreciate the time and effort you have taken to review our work, and we believe your suggestions have significantly improved the quality of the manuscript.

---

Q1. The parameter size of FCD.

A1. There are only several MLPs integrated in FCD and the parameter sizes of them depend on the hidden dimensions of the output of modality encoders.

Specifically, for each modality, CCD module has 4 MLPs with the input size $d^m$ and the output size $d^m$ ($\mathcal{F}^m\_{\text{h}}$, $\mathcal{F}^m\_{\text{s}}$, $\mathcal{F}^m\_{\text{URD-f}}$ and $\mathcal{F}^m\_{\text{URD-d}}$). So the parameter scale is $O({d^m}^2)$. For SDA module, there are M MLPs that map each unimodal feature to a common space and then map to the original dimension. Then parameter scale is $O(d^2+d\sum\_{m=1}^{M}d^{m})$.

Since the base methods have their unique designs, there is no exact parameter size ratio between FCD and base model. We report the training time cost in Appendix F, which may also reflect the parameter size relationship between the FCD and the original model. It can be seen that the growth rate brought by FCD is relatively large for models with original short training times. Conversely, the growth rate brought by FCD is relatively small.

Therefore, the FCD parameter scale is associated with the base model unimodal encoder output dimensions. If the output dimensions are large, the FCD parameter size would be larger than those whose output dimensions are small.

---

Q2. The measure-preserving bijection of URD and the ''measure'' in our paper.

A2. We technically apply SVD and construct skew symmetric matrix to constrain URD. We give the pseudo code to train FCD in Appendix B. Specifically, every singular value in $\Sigma$ is enforced to be greater than $10^{-5}$ to ensure that the weight matrix of the linear layer in URD remains invertible. Then, the skew symmetric matrix with exponential map is constructed to achieve the measure-preserving characteristic. Since the nesting of two measure-preserving functions is still measure-preserving, the URD is also a measure-preserving bijective function. Please refer to Appendix B for more details of the training process and further description of measure-preserving bijective URD.

We specify that URD should be ''measure-preserving'' to make Lemma A.1 in Appendix A hold. In Lemma A.1, if $\mathcal{F}$ is not measure-preserving, it may change the probability distribution of variables. For example, if $\mathcal{F}$ is not measure-preserving, let $Y=2X$ and $X$ is uniform distribution $P(X)=1/2$. In this case, the probability distribution of $Y$ should be $P(Y)=1/4$, Lemma A.1 do not hold. Therefore, the ''measure-preserving'' ensures that the probability distributions of variables are the same. And the ''measure'' here refers to the size of subsets within a stochastic variable's domain, allowing probability to be assigned to events.

---

Q3. Further elaboration of SCM (Fig. 1).

A3. As we stated in Introduction (Section 1) and Related Works (Section 2), previous researches mainly focused on better utilization of complementary and consistent information or mitigating aleatoric uncertainty within each modality. We summarize previous related works and empirically abstract the unimodal feature into 3 types of factor (synergistic, unique and redundant).

For example, in image-text task,

• The task-related parts are the ones that contribute to the final task prediction. The synergistic parts are both included in the image and the text modalities, which jointly contribute to the final task prediction.

• The unique parts are the semantic contents that only appear in image modality (or text modality) and don't appear in text modality (or image modality). Both synergistic and unique factors are task-related and derived from the task-related factor.

• The redundant parts are the uncertainty noise within one modality, such as the typos in text modality or quality of image, which is derived from the noise factor.

• The multimodal data store these information in their different format (the specific factor), but they are still associated in high-level semantic (the synergistic factor).

In case study (Section 5.5), we visualize the synergistic and unique parts in both image and text modality, which can more directly understand the corresponding regions of different types of features in the original sample.

---

Q4. The motivation and necessity of SDA.

A4. SDA module is used to constrain the extraction of synergistic feature from each modality. The synergistic feature represents the consistent information that presents in multiple modalities, facilitating alignment of modalities within a common representation space. Without SDA, there is no explicit strategy to constrain the features are able to be aligned within a common representation space, which causes the failure of synergistic feature extraction.

Similar to your idea, we also employed a shared MLP across all modalities ($\mathcal{F}\_{\text{SDA}}$, see Fig. 4). To further align the features, we propose $\mathcal{L}\_{\text{SDA}}$ (Eq. 15) to explicitly optimize $\mathcal{F}\_{\text{SDA}}$ and guide features from different modalities towards a common representation space. In our ablation study (Section 5.4), we set the coefficient of $\mathcal{L}\_{\text{SDA}}$ ($\lambda\_3$) equals to $0$ to validate the effectiveness of $\mathcal{L}\_{\text{SDA}}$, where only the parameter-sharing MLP is employed and there are no constrains on the synergistic information alignment. From Table 3, we can find that the performance decline without this explicit constraint. Therefore, it is necessary to explicitly supervise the module by $\mathcal{L}\_{\text{SDA}}$ (Eq. 15). Please refer to Section 5.4 for more details.

---

Additional comments:

Q1. Some typos: line 226 and 338

A1. Thank you again for your meticulous review. We will double check the details in the article.

---

Q2. The authors should more clearly and directly clarify what kind of claim or what kind of conclusion their theorem supports.

A2. Theorem 4.1 states that if the extraction function $\mathcal{F}^m\_{\text{mb}}$ is a measure-preserving bijection, then effectively extracting the task-related feature $\mathbf{u}^m\_n$ from the modality-specific feature $\mathbf{h}^m_n$ while separating out the modality noise $\mathbf{r}^m\_n$ via backdoor adjustment is equivalent to maximizing the mutual information between the annotation and the $\mathbf{u}^m\_n$ after causal intervention. Based on this, URD extracts the unique feature (u) via Eq. 6 where the unimodal noise is simultaneously removed.

---

Q3. The font of Definition should be different from the main contexts.

A3. Thank you for your suggestion. We will revise it in our camera-ready version if accepted.

---

Q4. For the case study, it is suggested to select and analyze challenging cases where prior methods proved ineffective. By demonstrating that the proposed approach succeeds in these scenarios—particularly when aligned with the SCM hypothesis—we can highlight its critical role.

A4. Thanks for your rigorous suggestion to the case study. We will conduct this experiment and revise it in our camera-ready version if accepted.

Thank you very much for your thoughtful and constructive comments, which would be greatly helpful to improve the quality of our work.

---

Q1. The standard deviation over several runs of our experiments.

A1. We conducted multiple independent experiments and found that each experimental result is basically the same. This is because we followed the hyperparameters reported in each base methods or their official scripts. And there is no stochastic mechanism designed in our approach. So the standard deviations are pretty small and they were not indicated in the paper. We are sorry for the unclear statement and will refine our description of the experimental design and results.

These are the standard deviations of 5 independent experiments on CMU-MOSI dataset.

---

Q2. The guarantee of separation between the s (synergistic) and h (specific) branches (purity).

A2. The guarantee of separation between the $\mathbf{s}$ and $\mathbf{h}$ branches is the direct optimization targets, i.e., the item $\mathcal{L}\_{\text{FCD}}$ (Eq. 17).

• (1) On the one hand, to separate the synergistic feature $\mathbf{s}$, $\mathcal{L}\_{\text{SDA}}$ (Eq. 15) is employed to optimize $\mathcal{F}\_{\text{s}}$ and SDA module. It constrains $\mathbf{s}$ to be mapped into a common representation space.

• (2) On the other hand, to separate the modality-specific feature $\mathbf{h}$, the other two optimization targets are employed, i.e., $\mathcal{L}\_{\text{Dis}}$ (Eq. 19) and $\mathcal{L}\_{\text{MI}}$ (Eq. 6). These two items jointly supervise the extraction of $\mathbf{h}$, ensuring it’s not only modality-specific but also task-related information.

Since the unique feature $\mathbf{u}$ is mixed in $\mathbf{h}$, there exists a complex high-level semantic overlap between $\mathbf{s}$ and $\mathbf{h}$, because $\mathbf{s}$ and $\mathbf{u}$ describe the same object (the task annotation is the same for a single data point). It is tricky to directly separate them since the semantic contents of different samples are various. If we directly add constraints (like orthogonality constraint), the two output may become task-related and task-agnostic. In this case, we cannot guarantee the task-related part contains both complementary and consistent information, which may cause information distortion.

Inspired by [1], we use two parallel MLPs to map the original feature to the synergistic $\mathbf{s}$ and the modality-specific $\mathbf{h}$ with the supervision of $\mathcal{L}\_{\text{FCD}}$, as we mentioned above. Although there are no regularization terms between these MLPs, the output of them are still constrained by $\mathcal{L}\_{\text{FCD}}$, ensuring them effectively extracting the corresponding features.

[1] Ying Zhou et.al. Triple disentangled representation learning for multimodal affective analysis. Information Fusion 2025.

---

Q3. The computational overhead of pairwise comparison L_SDA.

A3. Your concerns are very reasonable and worth improving upon. The purpose of designing this pairwise comparison loss function is to reduce the differences between any different modal distributions, in order to ensure that the extracted synergistic features are indeed shared among $M$ modalities. As you have pointed out, this strict constrain will cause the $O(M^2)$ complexity. To improve it, we can randomly sample two modalities in each iteration. Then only one Sinkhorn divergence between two distributions is calculated to achieve approximate alignment effect.

---

Q4. The sensitivity analysis of batch size hyperparameter.

A4. We reproduced and conducted our experiments based on the batch size given in each compared paper or their scripts. Since FCD is a plug-and-play module and we aim to validate the performance under the original experimental settings, we did not tune the batch size (or other base hyperparameters) in our experiments. Here, we conduct the sensitiveity analysis on Self-MM with FCD on the CMU-MOSI dataset and the results are as follows.

This experiment result shows that larger batch size may enhance the performance of the model because of the increment of negative sample diversity in contrastive losses. We would add the results to the camera-ready version if accepted.