CMoB: Modality Valuation via Causal Effect for Balanced Multimodal Learning

$Q1:$The global nature of ITE is mentioned in the paper, but the consistency under different modality combinations is not sufficiently verified.

$R1:$Thank you for your comment. The "global nature" means our individual treatment effect (ITE) evaluation applies to different modality combinations, but the ITE values for each modality vary across these combinations. We conducted four different modality combinations ：{T+A}, {T+V}, {V+A}, { T+A+V } for experimental verification using three modality datasets (CMU-MOSEI). The Concat method is a baseline method. The comparative experiments demonstrate the superiority and generalizability of our method, as shown in Table 1 .

Table1 Comparison with different modality combinations on CMU-MOSEI datasets. The evaluation metric is Accuracy (ACC).

$Q2:$Lacking ablation experiments to verify the effectiveness of each module.

$R2:$ Thanks for your comment. We conducted an ablation study to demonstrate the effectiveness of the proposed method on CREMA-D and KineticsSounds datasets , and the results are shown in the Table 2. Here, CQM represents our proposed a causal-aware modality contribution quantification method, and RE denotes the dynamic modality optimization Strategy. Ablation studies clearly demonstrated the critical contribution of our proposed modules to overall method performance enhancement.

Table2 Ablation analysis of proposed modules.

$Q3:$The specific implementation of the Mask(·) function in Equation 5 is not explicitly described.

$R3:$Thank you for your comment. We employed Time Masking for audio modality and Spatial Masking for video modality.

$Q4:$It is recommended to add efficiency tests on large-scale datasets and cross-modal number extension experiments.

$R4:$Following your suggestion, we conduct an experimental on large-scale dataset VGGSound. The VGGSound datasets consist of both audio and video modalities. The VGGSound dataset, which contains 310 classes and a wide range of audio events in everyday life, is a relatively large dataset. It includes 168,618 videos for training and validation, and 13,954 videos for testing.The experimental results show the superiority of our method. The results of the comparative experiment are shown in Table 3.

Table3 Comparison with different methods on VGGSound dataset.

In the manuscript we have already submitted, we have conducted experiments on more-than-two modalities datasets CMU-MOSEI and NVGesture. To further validate the effectiveness of our method under scenarios with more modalities, we follow this paper and conduct further experiments on the Caltech101-20 dataset [1]. We compare our method with the Concat method and the Shapley value method. The experimental results demonstrate the effectiveness of our method, as shown in Table 4.

Table4 Accuracy of our methods on Caltech101-20 dataset.

$Q5:$Can the quantification of causal effects explain the synergistic/competitive relationships between modalities? For example, the complementarity of audio-video modalities in emotion recognition.

$R5:$Thank you for your question. We can indeed explain synergistic/ competitive relationships between modalities by quantifying causal effects of each modality. We treat each modality as a treatment variable and the emotion label as the outcome. Using our defined benefit function B(M), we compute the causal effect of each modality. For example, when processing with audio–video modalities in emotion recognition:

• Synergistic: During iterative training, when a single modality cannot predict the emotion accurately, but adding the other modality achieves correct prediction, we have $ITE(audio/video)$=$B\bigl(\hat{F}\bigl(S(x)\bigr)\bigr) - B\Bigl(\hat{F}\bigl(S(x) \mid do(t = x^{(video/audio)})\bigr)\Bigr)$ = $B(multimodal) - B(video/audio)$ = $2$, indicating a synergistic relationship where audio and video are complementary.

• Competitive: During iterative training, if the audio modality alone successfully predicts the emotion label, yet adding the video modality yields an accurate prediction with lower confidence than that achieved using the audio modality alone. At this point, $ITE(video)$=$B\bigl(\hat{F}\bigl(S(x)\bigr)\bigr) - B\Bigl(\hat{F}\bigl(S(x) \mid do(t = x^{(audio)})\bigr)\Bigr)$ =$-1$, and the relationship between the two modalities is competitive.

When the causal effect of a modality is less than 1, it negatively impacts emotion prediction; if it is greater than 1, the impact is positive.

[1]Wei Y, Feng R, Wang Z, et al. Enhancing multimodal cooperation via sample-level modality valuation[C]. Proceedings of the Conference on Computer Vision and Pattern Recognition. 2024: 27338-27347.

$Q1:$How do the final performance figures change if temperature-scaled logits or negative log-likelihood are used instead of the default soft-max confidence?

$R1:$Following your suggestion, we replaced the default softmax confidence with temperature-scaled logits. We set the temperature parameter T=2 and T=0.5. The evaluation metric is Accuracy (ACC). The experimental results demonstrate that using temperature-scaled logits(T=2) is helpful for the target task. In our method, we use cross-entropy loss, which is essentially softmax + negative log-likelihood. Our method(CMoB) isn't restricted to using cross - entropy loss and softmax confidence. We mainly use them for a fair comparison with other methods.

$Q2:$For each benchmark, what is the percentage increase in wall-clock training time and peak GPU memory consumption relative to the strongest baseline?

$R2:$Thanks for your comment. Our modality selection and dynamic masking method incur higher computational overhead. We compare training time and computational overhead with the baseline model that performs best. Among them, the MLA method is the strongest baseline in the CREMA-D dataset, and the MMPareto method is the strongest baseline in the KineticsSounds dataset. Preliminary results show that our method increases GPU memory consumption by approximately 20% to35%. We consider that this overhead is acceptable given the improvement in the accuracy of the experimental results, as shown in the table below. Compared with methods that use data processing strategies (e.g., Shapley value) to address modality imbalance, our method achieves lower computational overhead and reduced training time.

$Q3:$Please report an ablation study that quantifies the individual gains contributed by each module of the proposed framework.

$R3:$Thanks for your comment. We conduct an ablation study to demonstrate the effectiveness of the proposed method on CREMA-D and KineticsSounds datasets, and the results are shown in the table below. Here, CQM represents our proposed a causal-aware modality contribution quantification method, and RE denotes the dynamic modality optimization Strategy. Ablation studies clearly demonstrate the critical contribution of our proposed modules to overall method performance enhancement.

$Q1:$The authors mention being inspired by the human nervous system but then shift to a causal learning perspective without clearly explaining the connection between these two concepts.

$R1:$ Thanks for your comment. Through analyzing the human cognitive system's processing of multimodal data, we observe that it first extracts discriminative causal features from raw multimodal inputs via experientially-guided mechanisms. Then, by using information entropy to dynamically quantify causal contributions across modalities, this process corresponds to the principle of "Causal Effect"in causal learning. Therefore, our framework simulates the cognitive system's " Dynamic Evaluation of Modality Contribution " by quantifying the causal effects at the sample level .

$Q2:$What are advantages of this approach over Shapley value or other similar techniques?

$R2:$Thanks for your comment. Shapley value method inevitably leads to exponentially longer training times and heavier computational burdens. Our method (CMoB) reduces this burden substantially, while providing theoretical guarantees through causal intervention principles. For example, when evaluating the contribution of the $M^3$ modality in a dataset containing three modalities {$M^1$, $M^2$,$M^3$}, the Shapley value method is necessary to calculate the combinations of four subsets {∅, $M^1$, $M^2$, $M^1$+$M^2$}. Our causal learning framework only calculates the {$M^1$+$M^2$} coalition. The peak GPU memory consumption on the CMU-MOSEI dataset was 19030MiB for CMOB method and 24384MiB for Shapley value method . Comparison experiments with the baseline of Shapley values[1] further demonstrate the superiority of our method, as shown in the table below.

Table1 Comparison with Shapley value algorithms. The evaluation metric is Accuracy (ACC).

$Q3:$The paper's writing style needs improvement, especially in the introduction, which spends excessive time discussing related work but fails to clearly explain the motivation behind the proposed method. In addition, Section 3.1 is too detailed and spends considerable space describing concepts like cross-entropy, which do not directly contribute to the core argument of the paper. This section could be more concise and focused.

$R3:$Thanks for your question. To address modality imbalance problem, current methods primarily rely on gradient variations to measure modality contribution. These methods primarily rely on modality-level contribution assessments to measure gaps in representational capabilities and enhance poorly learned modalities, overlooking the dynamic variations of modality contributions across individual samples. Although Shapley value-based methods can measure modality contributions at the sample level, they cause a sharp increase in computational overhead and training time on datasets with more than two modalities. Inspired by human cognitive science, we propose a causal-aware modality contribution quantification method from a causal perspective to capture fine-grained changes in modality contribution degrees within samples. And we dynamically select and optimize modalities based on real-time changes in their contributions.

In addition, In Section 3.1, we primarily formalize the modality imbalance problem. We will simplify section 3.1 and emphasize only components essential to modality imbalance understanding.

$Q4:$ In Section 4.3, the authors claim to replicate the experiment from [45] on the "scarcely informative modality" case but using the original CREMA-D dataset. However, [45] created such a case by adding significant noise to the audio modality of the CREMA-D dataset. The original CREMA-D dataset does not inherently contain "scarcely informative modality" cases.

$R4:$Thanks for your comment. Here we may not have provided a clear expression. We modify the audio data of the CREMA-D dataset, adding extra white Gaussian noise to make it noisier and scarcely discriminative. In this section, we conduct comparative experiments using the processed CREMA-D dataset.

$Q5$:The quality of the images in the paper, particularly Figures 2 and 3, is poor. The legends in these figures are difficult to read, which diminishes the clarity of the data presented.

$R5:$Thanks for your comment. We acknowledge that the current presentation does not adequately convey critical details. The legends in these figures are difficult to decipher because the annotation elements were not scaled appropriately. In the next version, we will use a higher resolution (600 dpi) and adjust the font size of all annotations to 10pt+, and increase the line thickness by 50% in Figures 2 and 3.

[1]Wei Y, Feng R, Wang Z, et al. Enhancing multimodal cooperation via sample-level modality valuation[C]. Proceedings of the Conference on Computer Vision and Pattern Recognition. 2024: 27338-27347.

$Q1:$ Some important references are missing. The sample-level modality contribution was discussed in "Cao, et al: Predictive Dynamic Fusion. ICML 2024".

$R1:$ Thanks for your comment. Although both our method and the PDF[1] method are sample-level modality contribution quantification, the two methods focus on different aspects. Our method(CMoB), using causal learning and intervention-based contribution estimation, offers better interpretability. CMoB focuses on identifying dominant/ weak modalities at the sample-level during training, whereas PDF focuses on assessing quality disparities among modalities. CMOB emphasizes enhancing learning for weak modalities, whereas PDF aims to reduce reliance on low-quality modalities via Relative Calibration (RC). We conduct comparative experiments with the PDF method on the CREMA-D dataset. The experimental results show that our proposed method exhibits strong robustness, as shown in the Table1. We will discuss Cao et al (ICML 2024) in the Related Work section and include experimental comparisons in the revision.

$Q2:$ The authors should compare some relatively new algorithms.

$R2:$ Following your suggestion, we conduct comparative experiments with the latest modality imbalance algorithms on the CREMA-D dataset, as shown in the Table 1. To further validate our approach, we add Gaussian noise on 50% modalities and ϵ presents the noise degree. The experimental results show the superiority of our method. In our manuscript , we also compared with relatively new algorithms, such as the MMPareto, MLA, Relearning, and MMCooperation methods were proposed in 2024, and the MMCosine, PMR, and AGM methods were proposed in 2023.

Table1 Comparison with the latest algorithms on the CREMA-D dataset. The evaluation metric is Accuracy (ACC).

$Q3:$ The language quality of the submission is not very good.

$R3:$ Thanks for your comment. We will comprehensively revise the manuscript to improve its clarity, grammatical accuracy, and conciseness in the next version.

[1] Cao B, Xia Y, Ding Y, et al. Predictive Dynamic Fusion[C].Proceedings of the International Conference on Machine Learning. PMLR, 2024: 5608-5628.

[2] Wei Y, Feng R, Wang Z, et al. Enhancing multimodal cooperation via sample-level modality valuation[C]. Proceedings of the Conference on Computer Vision and Pattern Recognition. 2024: 27338-27347.

[3]Wei Y, Hu D, Du H, et al. On-the-fly modulation for balanced multimodal learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025:47(1):469–485.

[4]Han W, Cai C, Guo Y, et al. Erl-mr: Harnessing the power of euler feature representations for balanced multi-modal learning[C]. Proceedings of the 32nd ACM International Conference on Multimedia. 2024: 4591-4600.

[5]Zhou Y, Liang X, Xu Y, et al. Dataset-aware Utopia modality contribution for imbalanced multimodal learning[J]. Information Fusion, 2025: 103383.