Balancing Multimodal Training Through Game-Theoretic Regularization

We would like to thank the reviewer for the thoughtful comments.

We agree that several of our figure and table captions were overly verbose, as we initially aimed to make them fully self-contained. Following your suggestion, we have now revised the captions throughout the paper to be more concise and focused, emphasizing the key takeaways without overwhelming the reader. Regarding the integration of Table 1 with the introduction, we understand the reviewer’s concern to be that the table did not clearly reinforce the core motivation of the paper, namely, the presence of multimodal competition in standard training settings. We have now clarified this in both the introduction and the caption of Table 1, explicitly highlighting how Joint Training underperforms relative to the Ensemble baseline, which provides early empirical evidence of competition between modalities. These changes aim to better guide the reader in connecting the table to the central problem addressed by MCR.

Regarding the error matrices figure, we agree that the original figure lacked clarity in conveying what constitutes a “better” result and how it connects to synergy and routing. To address this, we have revised the caption to explicitly state that a good result corresponds to low percentages in the top row (multimodal failure) and high percentages in the bottom row (multimodal success). We have also visually annotated with boxes the first CREMA-D ResNet matrix to highlight the "Both Wrong" column as evidence of synergy and the remaining columns as routing. These additions make the connection to our discussion more explicit and convey some of the key messages more clearly.

Regarding the interesting question about Eq. 12, we agree that our explanation in the manuscript was insufficiently detailed. Whether the gradient of the $L_{MIPD}$ with respect to the perturbed modality provides useful or detrimental signal depends critically on the type of perturbation applied. When the perturbation corresponds to predefined or out-of-distribution inputs, such as zero-masking or adding noise, the resulting gradients may encourage the model to fit spurious patterns, potentially harming robustness. In contrast, when the perturbation is implemented via within-batch permutations (as in our main experiments), the perturbed modality remains in-distribution, and the loss $L_{\text{MIPD}\_1}$ can have a constructive influence on $\theta_1$. The formulation is symmetric with respect to $L_{\text{MIPD}\_1}$ and $L\_{\text{MIPD}\_2}$ under permutation-based perturbations under the JSD, but this symmetry breaks under arbitrary or task-information removal augmentations (e.g., those that remove class-relevant information entirely). In Appendix A.10, we explore such perturbation types and clarify that applying both $L\_{\text{MIPD}}$ under these conditions may degrade model robustness by reinforcing spurious or misleading input patterns. It is worth noting that removing the $L_{\text{MIPD}}$ terms for these alternative perturbation strategies did not lead to improved performance. We have now added this clarification to the manuscript to better contextualize the behavior of $L\_{\text{MIPD}}$ under different perturbation regimes.

We have also incorporated the reviewer’s suggestion to evaluate zero-masking as a perturbation method for estimating conditional mutual information. While we had previously considered zero-masking in the input space, particularly in the context of computing Shapley values, we had not extended this approach to the latent space. In response, we have now expanded our ablation study (Appendix A.10) to include zero-masking in the latent space, which we find provides similar computational advantages to the permutation-based method. The corresponding results are shown in the table below.

Regarding Equation 10, while the proof indicates a lower bound that encompasses $I(X_1; Y \mid X_2)$ and $I(X_2; Y \mid X_1)$, it does not provide control over these two terms. The MIPD components introduced in our formulation make these conditional dependencies explicit and separable, enabling us to modulate their relative influence during training. This separation allows for flexible strategy design in how modality interactions are encouraged or discouraged. Adjusting this balance can lead to performance gains beyond simply maximizing each individual term, as shown in the corresponding ablation study (Appendix A.8). The MIPD formulation thus introduces control over inter-modal behavior that would otherwise be entangled only as a joint lower bound in the objective of Equation 10.

Finally, we thank the reviewer for pointing out the notational inconsistency. We intended to describe the parameter updates for each encoder separately, rather than for the joint parameter set. To clarify this, we have rewritten the gradient expression as:

$\nabla\_{\theta\_1} \mathcal{L}\_{MIPD} = \lambda\_M ( \nabla\_{\theta\_1}\mathcal{L}\_{MIPD\_1} + k \nabla\_{\theta\_1}\mathcal{L}\_{MIPD\_2} )$

$\nabla\_{\theta\_2} \mathcal{L}\_{MIPD} = \lambda\_M ( \nabla\_{\theta\_2}\mathcal{L}\_{MIPD\_2} + k \nabla\_{\theta\_2}\mathcal{L}\_{MIPD\_1} )$

This updated notation should resolve the ambiguity and align the gradients with the individual parameter sets.

We appreciate the reviewer’s observation regarding cross-modal complementarity and we agree that this is a valid and important limitation. As noted in our discussion (Section 5) and detailed in the error analysis (Section 4.3, Appendix A.11), MCR excels at routing information when at least one modality provides a reliable signal. However, its ability to capture synergetic gains, particularly when all unimodal branches fail, is currently limited.

Our initial hypothesis was that encouraging each modality to influence the fused prediction would inherently promote synergetic behaviour. However, the results suggest that emergent synergy does not arise automatically from this dynamic. We view this as an open challenge and an exciting direction for future work. Specifically, we are exploring extensions that encourage the joint representation to encode complementary information beyond what each unimodal pathway contributes individually, e.g., via terms like $I(f(x_1, x_2); Y \mid f(x_1), f(x_2))$. Unlocking full cross-modal complementarity, especially in difficult cases where all modalities are individually weak, remains an open frontier. We thank the reviewer for highlighting this and agree that addressing this challenge is essential for further advancing multimodal learning.

Regarding the reviewer’s suggestion for additional evidence on generative problems. We agree that evaluating multimodal training methods on generative tasks would broaden their empirical scope. It is important to clarify that our method is designed for settings where multiple modalities are available as inputs. In contrast, tasks like text-to-image or image-to-text generation typically involve only a single input modality, trying to generate the second one, and thus do not align with the central premise of MCR, which is to regulate the interaction and competition between input modalities during training. Applying MCR to such generative tasks would require a reformulation of the framework. We agree that extending MCR to generative tasks involving multiple conditioning modalities (e.g. audio + image → caption) would be a compelling next step. In such setups, managing how each modality influences the generation process could benefit from the same principles that underlie MCR. However, evaluating training strategies in generative settings remains challenging, particularly given the lack of established benchmarks for comparing multimodal training dynamics in these tasks. Additionally, many of the baseline methods we compare against are not directly applicable in those settings without major adaptation. For these reasons, we focused on classification and regression tasks where (1) multiple input modalities are jointly present, and (2) fair, well-established evaluation benchmarks for multimodal balancing exist. We appreciate the reviewer highlighting this direction, and we believe it represents an exciting opportunity for extending our framework in future work.

We thank the reviewer for the suggestion on exploring the different loss terms during training. Indeed, analyzing the evolution of each component of the MCR loss (i.e., the two MIPD terms, the contrastive loss, and the CEB term) could provide valuable insights into the learning dynamics and modality interactions throughout training. In our experiments, we do track these components individually. For instance, we observe that the contrastive loss typically decreases smoothly, while the two MIPD terms alternate in which modality dominates at different points during training, often reflecting shifts in the modality that contributes more to the fused output. However, these trends do not necessarily remain smooth during training, and interpreting them in a systematic and generalizable way has proven non-trivial. Furthermore, leveraging this information to explicitly infer synergy or competition among the modalities would require extending our ways of evaluating the multimodal models. We recognize that such loss-term dynamics analysis is a meaningful step toward deeper understanding and calibration of multimodal learning behavior. In response to the reviewer’s suggestion, we have now included an additional paragraph in the discussion section acknowledging the value of this direction and outlining our intent to pursue it in future work. We appreciate the reviewer highlighting this important perspective.

Furthermore, the thoughtful conjecture regarding the greedy strategy in Section 3.2 is well taken. Our current implementation promotes competition between modalities (k = −1), which indeed prioritizes maximizing each modality’s contribution, even at the cost of the other. While this improves routing (as shown in “one correct” cases), it may hinder synergetic behaviour, especially when no single modality suffices on its own. To examine this, we conducted a further error analysis of the different gaming strategies on the "both wrong", and we observed that there was no difference among the three game strategies.

Our initial hypothesis was that promoting each modality to influence the output would naturally encourage collaboration between them, thereby enhancing synergetic behavior. However, our empirical results did not support this assumption. We acknowledge in the discussion of the paper that there is no term to promote such synergy, and we are currently working on improving this aspect of the framework. We agree that this limitation highlights an important avenue for further discussion and future research.

Lastly, thank you for bringing to our attention the recent work "Can One Modality Model Synergize Training of Other Modality Models?". We have carefully examined it and find that it tackles a complementary but distinct problem. Their focus is on knowledge distillation across modalities, particularly how a modality can aid the training of another, and especially in unpaired data settings. This is indeed an impressive contribution to cross-modal representation learning, but it differs fundamentally from our goal. In contrast, our work addresses how to regulate modality interactions within joint multimodal training, and how these dynamics affect the performance of the fused model, rather than improving solely the unimodal branches. Nevertheless, there is a growing body of work that aims to quantify modality interactions, and we have drawn inspiration from several of these efforts, e.g., Liang et al. “Quantifying & Modeling Multimodal Interactions”. To evaluate their estimation of multimodal interactions, they rely on human-annotated ground truth regarding the informativeness of each modality, which is typically not available. In contrast, our objective was to develop a method that is modality- and task-agnostic, and thus we deliberately avoided reliance on such supervision. That said, we recognize the promise of this direction, and in future work, we aim to incorporate calibrated measures of uniqueness, redundancy, and synergy. These tools could offer deeper insight into the behavior of multimodal systems.

We thank the reviewer for pointing this out the absence of standard deviation in Table 1 for MOSI, MOSEI, and Something-Something was due to those results being reported from a single run. We have now addressed this by rerunning our training pipeline on each method three times, using the same dataset splits (as provided by each benchmark). The corresponding dataset descriptions have been updated to reflect this protocol and ensure full reproducibility. The revised results, including standard deviations, are shown below:

Thank you for the suggestion on the statistical testing, we have now run Wilcoxon and Holm correction on the paired comparison of our method with each one based on the different model/dataset averaged performance. We conclude that one D&R that achieves better on AVE-Conformer produces non-significant p values. The comparisons are included in a new appendix section with table as shown below:

We have now performed Wilcoxon signed-rank tests and Holm corrections to compare MCR against each baseline across the averaged results for each combination of dataset/model. Compared to the rest of the multimodal training methods, we see that MCR is statistically significant, except for D&R, which outperforms MCR on the AVE-Conformer setup. We have also conducted separate statistical tests on Tables 3 and 5, but we did not find the variations to be significant under the given tests. The full results are presented in the table below and included in a new appendix section.

We have performed Wilcoxon signed-rank tests with Holm correction to compare MCR against each baseline across the averaged results for each dataset/model combination. MCR shows statistically significant improvements over all multimodal training methods, except for D&R, which outperforms MCR on one of the setups, namely the AVE-Conformer. We note, however, that the number of evaluation points (9 in total) is relatively limited, which constrains the strength of such statistical claims. Despite that, our results are shown in the table below and included in a new appendix section.

For Tables 3 and 5, the ablation studies were conducted on a subset of the dataset/model combinations, which does not provide enough samples to support reliable statistical testing. Once the ablations are completed across all 9 datasets, we plan to include statistical significance testing in the camera-ready version to ensure consistency with our main results.