Multimodal Negative Learning

We sincerely thank the reviewer for the valuable comments, the recognition of our method’s novelty and theoretical contributions, and the acknowledgment of its potential benefits to broader multimodal learning research. We believe the constructive feedback will help us further improve the paper and enhance its impact on the community.

W1 The authors conducted extensive experiments, however, the competing method “LATE FUSION” is confusing for me. QMF and PDF also belong to late fusion methods. Therefore, the claimed “LATE FUSION” is not inappropriate.

Thanks for your comment. The "LATE FUSION" mentioned here specifically refers to "STATIC LATE FUSION." We have corrected this in the manuscript and will ensure the final version reflects the change.

W4 & Q1 It is reasonable to consider the dynamic modality dominance in multimodal learning, while a more clear illustration will be helpful. & What’s the key effect of dynamic guidance on multimodal negative learning? Eq. 6 should be further explained.

Thanks for your question. The dynamic guidance mechanism is indeed a critical component of MNL. Its core is defined in Definition 3.3, which jointly considers both $P _y$ and the Unimodal Confidence Margin (UCoM) to identify the Robust Dominant Modality (RDM) and the Inferior Modalities (IMs). Specifically, the RDM is selected such that its $P _y$ and UCoM are both greater than those of the IMs, thereby enabling the guidance direction to be determined dynamically.

Regarding Eq. (6), its key function is to determine two aspects: the guidance direction, which involves identifying the RDM and IMs (as described in Definition 3.3, with Eq. (6) focusing on the two-modality case), and the guidance scope, which is restricted to non-target classes. If dynamic guidance is absent in multimodal negative learning:

• A modality that predicts incorrectly might guide a modality that predicts correctly. Even if the guidance is restricted to non-target classes, this can still disrupt the optimization process of the modalities.
• A modality with a low Unimodal Confidence Margin (UCoM) might guide a modality with a high UCoM, which can potentially harm the robustness lower bound of the multimodal system.

W5 Since this work can be flexibly integrated with existing methods, I suggest the authors present the pseudo code in supplementary material.

Thank you for the excellent suggestion. Below, we provide the pseudocode of the algorithm. The inclusion of this pseudocode will be reflected in the final version of the paper.

Algorithm 1: Multimodal Learning with MNL

Input:
 Training set D_train = {(x_i, y_i)}_{i=1}^N ⊂ X × Y
 Iteration number T, warm-up epoch W

for t = 1 to T do
    Sample a fresh mini-batch B_t from D_train
    Feed-forward the batched data B_t to the model
    Obtain fusion logits and modality-specific logits from the model

    if t > W then
        Identify RDM and IMs according to Definition 3.3
        Compute MNL loss according to Eq. (5)
        Compute total loss according to Eq. (7)
    else
        Compute cross-entropy loss
    end if

    Update model parameters
end for

Q2 Is there any trade-off between multimodal positive learning and multimodal negative learning in Eq. 7?

Thanks for your question. We present our results on the MVSA dataset, where the CE weight is fixed at 1 and the MNL weight is varied from 0.2 to 2.0 under different noise levels.

Similar results are observed across multiple datasets. After careful consideration, we set the weights of both losses (CE and MNL) to be equal at 1.

Q3 There are two stages during the training phase. How to determine stages 1 and 2 in practice? More details should be provided.

Thank you for your question. We can provide the table (CREMA-D dataset, Noise = 0).

We observe that the hyperparameter warm-up epoch is not sensitive (we set it to 10). This is because the benefit of MNL is relatively small in the early stages of training, and becomes more effective as training progresses and the modality gap becomes more pronounced, allowing for more meaningful guidance. Therefore, the warm-up is primarily introduced to reduce unnecessary computational cost during the initial training phase.

Q4 Although the proposed method can be integrated with existing methods without introducing additional inference overhead, the additional training cost should be reported.

We are glad to report the additional training cost. Specifically, we calculate the average time per training iteration (per batch), with batch sizes of 16, 16, 32, and 64 for MVSA, FOOD101, NYUDv2, and CREMAD, respectively.

These results indicate that the additional overhead introduced by MNL is minimal.

presentation issues

Thank you for pointing out the typo. We have revised the description of “LATE FUSION,” improved the clarity of the figures and tables, and made the equations more comprehensible. Finally, we have carefully reviewed and corrected all representation issues throughout the paper.

We'd like to thank the reviewer for the valuable comments and the recognition of our novel “Learn Not to Be” paradigm. We appreciate your support and constructive suggestions, and we address your concerns in detail as follows.

W1 In table 1 and table 2, the improvement from the baseline is not consistently strong. For instance, when MNL is applied to state-of-the-art dynamic fusion methods like PDF and QMF, the performance gains are often marginal, sometimes less than 1%.

Thank you for your valuable comment. Taking PDF [1] as an example, PDF aims to effectively leverage multimodal data, whereas MNL focuses on enhancing the weak model's perceptual ability. Therefore, when we follow the standard setting and directly integrate the two methods, their respective advantages may not be fully exploited. For dynamic fusion methods such as PDF, we suggest a two-step training procedure: First, use MNL to enhance the model’s recognition ability, especially for the weaker modality (Step 1), and then apply PDF to dynamically fuse informative content from different modalities (Step 2).

Specifically, during training:

• Step 1: We perform multimodal fusion based on static late fusion and apply the MNL, while simultaneously training PDF to learn the ability to predict fusion weights $w$.
• Step 2: Based on the model obtained in Step 1, we utilize PDF to predict $w$ and conduct dynamic fusion accordingly.

To validate this setting, we have conducted additional experiments on the MVSA dataset under Gaussian noise with varying levels of noise (ε = 0, ε = 5, ε = 10). The results are summarized in the following table (over 5 seeds).

The most notable improvement occurs when ε = 5, where the performance of MNL (step1) + PDF (step2) surpasses that of PDF by 0.42%, in contrast to the 0.33% drop observed with PDF + MNL. The MNL (step1) + PDF (step2) configuration not only enhances the performance of the weaker modality but also dynamically allocates weights at inference based on data quality, thereby delivering a more effective boost to overall model performance.

W2 Theorem 3.1 assumes a statis fusion weights, yet experiments include dynamic fusion, creating a gap between theory and empirical results.

Thanks for your comment. MNL supports dynamic fusion, but according to Eq. (4), the dynamic weight $w$ is expected to be positively correlated with $\xi$. However, since $\xi$ is difficult to predict in practice, it becomes challenging to adjust the weights accordingly in a truly dynamic manner. Therefore, we adopt a static fusion assumption.

As indicated by Eq. (4), when $w$ is fixed, the robustness lower bound improves more steadily; when $w$ is dynamic, the robustness gain varies depending on how the weights are set, which may influence the overall performance improvement. This also explains why the benefit of combining our method with existing dynamic fusion approaches such as PDF is somewhat limited.

To mitigate the influence of dynamic weighting on MNL, we first training a static multimodal fusion system using MNL to enhance the model’s ability to perceive weak modalities, and then applying a dynamic fusion strategy like PDF to further boost overall performance (MNL (step1) + PDF (step2)).

W3 Equation 5 assembles knowledge distillation, where the "soft labels" from a teacher (RDM) are used to train a student (IM), diminishing the novelty of the paper.

Thanks for your valuable comment. We propose a new MNL learning paradigm inspired by knowledge distillation (KD).

Differently, KD focuses on training a student model under the guidance of a fixed teacher model, whereas our method focuses on dynamically enhancing the model’s ability to recognize non-target classes of the weak modality by leveraging the dominant modality.

In addition, beyond the commonly used KL divergence in KD, we also explore using L1 and L2 losses to implement MNL. We believe that our MNL has the potential to be compatible with more collaborative learning paradigms and adaptable to a wider range of tasks.

To explore this, we extended our experiments by modifying the MNL loss formulation (Eq. (5)) and implemented negative learning using L1 or L2 losses. Table 1 MVSA dataset under Gaussian noise (over 5 seeds)

Table 2 MVSA dataset under Salt noise (over 5 seeds)

Q1 What are the loss weights in Eq. 7 (is MNL summed equally with CE losses?)

Yes, according to our setup, the MNL loss is summed equally with the CE loss. To demonstrate this, we present our results on the MVSA dataset, where the CE weight is fixed at 1 and the MNL weight is varied from 0.2 to 2.0 under different noise levels.

Similar results are observed across multiple datasets. After careful consideration, we set the weights of both losses to be equal at 1.

Q2 What are the criteria for transitioning Stage 1→Stage 2 training?

Thank you for your question. We can provide the following table (CREMA-D dataset, Noise = 0).

We observe that the hyperparameter warm-up epoch is not sensitive. We set it to 10.

Q3 Fig 1(b) shows forced alignment harms weak modalities. Does MNL mitigate such collapse? It will be beneficial to provide examples where MNL fails to preserve weak-modality advantages, and quantify how often this occurs.

Thank you for your question. First, regarding whether MNL can mitigate such collapse, the following table presents a comparison of MNL (warm-up epoch = 10) and KL Guidance in terms of the number of Audio-wrong but Video-right samples across different epochs.

It is evident that MNL consistently results in significantly fewer audio-wrong but video-right samples compared to KL Guidance at the same training epoch.

• In KL Guidance, the audio modality (which generally performs better) is used to guide the video modality even when its own prediction is incorrect and the video prediction is correct. This misalignment introduces a conflict between the cross-entropy loss and the KL divergence, often leading to optimization collapse.

• In contrast, MNL dynamically adjusts the guidance direction based on Definition 3.3, thereby avoiding such conflicts.

As for cases where MNL fails to preserve weak-modality advantages, we take the MVSA dataset as an example. When applying MNL, the guidance direction is determined dynamically based on $P_y$ and $\xi$(UCoM), and according to Eq. (6), the guiding modality must have both higher $P_y$ and $\xi$ than the guided one. However, this condition is not always satisfied for every sample. Out of 1,555 training samples in MVSA, we found 143 samples where MNL was inactive (9.196%). Among these cases, it is possible that weak-modality advantages existed but were not preserved.

[1]Predictive dynamic fusion. ICML2024

We sincerely thank you for your valuable comments and appreciate your recognition of our proposed paradigm "Learning not to be" through Multimodal Negative Learning (MNL) and the theoretical foundation based on UCoM. We believe your constructive feedback will help improve the paper and enhance its potential impact on the multimodal learning community.

W1 & Q1 The two-stage training scheme (standard CE followed by MNL) adds some complexity to training dynamics and might not be ideal for scenarios requiring fast adaptation. & Training complexity analysis.

Thanks for your comment. Integrating MNL with baseline methods introduces minimal computational overhead and does not require any modification to the baseline network architecture. During training, MNL operates only at the output level by identifying the Robust Dominant Modality (RDM) and the Inferior Modality (IM), determining the direction of guidance, and computing the additional loss. During inference, the process remains identical to the baseline and does not introduce any extra cost.

We are glad to report the additional training cost. Specifically, we calculate the average time per training iteration (per batch) on MVSA, FOOD101, NYUDv2, and CREMAD, respectively.

These results indicate that the additional overhead introduced by MNL is minimal.

W2 It is recommended that the authors provide a more detailed algorithmic workflow or pseudocode to facilitate understanding and reproducibility of the proposed method.

Thank you for the excellent suggestion. Below, we provide the pseudocode of the algorithm. The inclusion of this pseudocode will be reflected in the final version of the paper.

Algorithm 1: Multimodal Learning with MNL

Input:
 Training set D_train = {(x_i, y_i)}_{i=1}^N ⊂ X × Y
 Iteration number T, warm-up epoch W

for t = 1 to T do
    Sample a fresh mini-batch B_t from D_train
    Feed-forward the batched data B_t to the model
    Obtain fusion logits and modality-specific logits from the model

    if t > W then
        Identify RDM and IMs according to Definition 3.3
        Compute MNL loss according to Eq. (5)
        Compute total loss according to Eq. (7)
    else
        Compute cross-entropy loss
    end if

    Update model parameters
end for

W3 & Q2 While MNL is presented for dual-modality cases, its extension to three or more modalities (especially when all are noisy) is not discussed or demonstrated.

We conducted additional experiments on the CMU-MOSEI dataset[1], which contains three modalities: text, audio, and visual. In this setting, the Robust Dominant Modality(RDM) is used to guide the other two weaker modalities. The RDM is identified by comparing $P_y$ and $\xi$ (UCoM) across modalities, in accordance with Definition 3.3. This strategy naturally generalizes to scenarios involving more than two modalities.

Table 1 presents the experimental results conducted on the CMU-MOSEI dataset. Gaussian noise for visual modality, blank noise for the text modality, SNR-based noise for audio modality

Table 2 ablation experiments

Thank you again for this valuable suggestion. We have added the experiment to the manuscript, and it will be included in the final version.

[1]Multibench: Multiscale benchmarks for multimodal representation learning. Advances in neural information processing systems 2021

We sincerely appreciate the reviewers’ positive recognition of the novelty of our “Learn Not to Be” concept, the soundness of the theoretical justification. Below, we provide detailed responses to your concerns.

W1 lack of proof in which conditions modality imbalance exists and how serious it is. In MLLM, it’s also common to sometimes rely on one modality but sometimes rely on the other.

Thank you for your valuable comment.

Multimodal imbalance arises from the model’s limited ability to perceive and utilize weak modalities [1], rather than deficiencies in the input data itself [2].

• Multimodal imbalance often occurs during model optimization [3], where the learning of one modality suppresses the learning of others.
• Multimodal imbalance is not related to the unbalanced usage of different modalities, nor does it imply that certain modalities provide less informative data.
• Even if a modality itself is informative, the model may still fail to make accurate predictions if it cannot effectively exploit the modality’s useful information.

Please kindly note that this notion does not conflict with dynamic fusion strategies that utilize modalities based on data quality.

In our paper, using the CREMA-D dataset as an example, the audio modality alone achieves 60.70% accuracy, and the visual modality 56.23%.

• With static late fusion and joint training, audio accuracy drops slightly to 59.35%, while visual accuracy decreases sharply to 41.12%, demonstrating severe modality imbalance during optimization.
• However, with MNL guidance, fusion accuracy improves to 73.71%, and the visual modality’s accuracy increases to 58.05% (>56.23%), surpassing its standalone performance.

W2 This paper also claims MNL can preserve specific information for weak modalities, but there is lack of such proof or analysis as well.

Thanks for your valuable comments. We have discussed the specific information preservation for weak modalities in Line 298-313 and Figure 4 (a), Line 51-72 and Figure 2 in Supp.

As shown in Figure 4(a), we compare the average KL divergence between modality predictions and the corresponding performance under different guidance strategies and noise levels.

• All-Class guidance reduces KL divergence by forcing the weaker modality to imitate the stronger one, but this leads to over-alignment and performance degradation.
• In contrast, MNL applies guidance only on non-target classes, resulting in higher KL divergence and consistently better performance.

MNL achieves improvements of 1.83%, 0.85%, and 4.64% under increasing noise levels. These results indicate that preserving the weaker modality’s distinct information improves robustness and cross-modal complementarity.

In Supplementary B.1 (shown as pictures),Follow [4] we employ the dit toolkit [5] to perform partial information decomposition, aiming to quantify the unique information contributed by each modality. Furthermore, we present a quantitative analysis to support our findings:

Table 1 under Gaussian noise

Table 2 Under Salt Noise

As shown in the results, MNL enhances the consistency of weak modality (visual) in expressing its unique information and leads to improved performance. This provides direct evidence and analysis that MNL effectively preserves specific information for weak modalities.

W3 & Q2 this paper are all for multimodal classification. It will be more convincing if this paper can show improvement on common MLLM tasks. & the paper’s name looks too general

Thank you for your valuable comment.

• Considering that most existing MLLM models [6] primarily perform feature-level fusion, to demonstrate the generalizability of our approach,we extend MNL to large language models, forming a multi-LLM fusion setting.

  Specifically, on the MathQA dataset [11], a choice math reasoning benchmark, we fine-tune InternVL2_5-1B and Qwen2.5-1.5B (which share the same vocabulary) using LoRA. Furthermore, during training, we apply MNL by masking the correct option and using $P_y$ and UCoM to allow the stronger model to guide the weaker one.

  We compare the performance of STATIC LATE FUSION and STATIC LATE FUSION + MNL to validate the effectiveness of our method. As shown in the table, applying MNL consistently improves both the individual LLM and the fused result.

• MNL belongs to multimodal learning. Multimodal learning is indeed a sophisticated and widely studied paradigm in the machine learning community and it refers to the integration and joint modeling of data from different modalities such as text, images, speech, video[7]. The aim is to improve a model's understanding capability and generalization performance[8]. This paradigm naturally covers a wide range of tasks, including multimodal classification [9,10], all of which fall under the broad definition of multimodal learning.

  the core of our method lies in establishing a negative learning paradigm within a multimodal system. Specifically, the Robust Dominant Modality (RDM) provides supervision on non-target classes to guide the Inferior Modalities (IMs). This strategy not only preserves the unique information inherent to the IMs but also enhances the overall robustness of the multimodal system. Following the reviewers' suggestion, we further validated the effectiveness of MNL in the multiple-LLM setting.

  Therefore, we believe that naming our method Multimodal Negative Learning is both appropriate and representative of its core contribution.

Q1 Do we have any ablations to the warm-up settings (e.g. steps)? This looks to affect the final model performance much. What is the additional efficiency cost for MNL training?

Thank you for your question.

(1) Warm-up Settings:

We have provided the ablations on warm-up epoch. According to the following table, the hyperparameter warm-up epoch is not sensitive in practice (we set it to 10). The reason is that the benefit of MNL is relatively limited during the early stages of training. As training progresses and the modality gap becomes more evident, MNL provides more effective guidance. Therefore, the warm-up setting primarily serves to reduce unnecessary computational cost in the early phase of MNL training.

(2) Efficiency Cost:

We provide a detailed efficiency analysis of MNL training’s extra cost by reporting average training iteration time on MVSA, FOOD101, NYUDv2, and CREMA-D, with no added overhead during inference.

These results indicate that the additional overhead introduced by MNL is minimal.

Limitations are not mentioned in this paper.

Thanks for your comment. we have provided a detailed discussion of the limitations Line 123-142 in Supplementary E. For example, the additional computational overhead introduced by MNL during training.

presentation issues

Thank you for pointing this out. We have corrected the typo in the legend of Figure 1(b) and carefully reviewed and revised all representation issues throughout the paper.

And we sincerely hope that our responses have sufficiently addressed your concerns. Should any questions remain, we would be more than glad to provide further clarification. If you find our clarifications satisfactory, we would greatly appreciate your consideration in updating your score to support the acceptance of our paper.

[1]Balanced multimodal learning via on-the-fly gradient modulation. CVPR2022

[2]Missing modality imagination network for emotion recognition with uncertain missing modalities. ACL2021

[3]Balanced Multimodal Learning via On-the-fly Gradient Modulation. CVPR2022

[4]Reconboost:Boosting can achieve modality reconcilement. ICML2024

[5]dit: a Python package for discrete information theory.

[6]Mm-llms: Recent advances in multimodal large language models. ACL2024

[7] Multimodal machine learning: A survey and taxonomy.TPAMI2018

[8]A theory of multimodal learning. NeurIPS2023

[9]Trusted multiview classification. ICLR2021

[10]Multimodal dynamics: Dynamical fusion for trustworthy multimodal classification. CVPR2022

[11]Mathqa: Towards interpretable math word problem solving with operation-based formalisms.

Thanks for your valuable feedback, which greatly contributed to improving the paper!

W1: Enhance target class to supress non-target class. & Compare with positive learning.

Thanks for your comment. There maybe some potential misunderstanding. MNL aims to guide the weak modality’s predictions on non-target classes using the dominant modality’s predictions on non-target classes, rather than constraining the weak modality’s prediction on the target class through the dominant modality’s prediction on the target class. We have added experiments on target class guidance (positive learning) and non-target classes guidance (MNL). As shown in the Table below, the positive learning is not effective enough.

(1) Guidance on target class: Using the dominant modality's prediction on target class to guide that of the weak modality essentially performs single-class guidance, which is similar to applying a soft-label constraint on the target class. This overlaps with the supervision already provided by the cross-entropy loss, thus offering limited additional benefits.

(2) Guidance on non-target classes: By leveraging the dominant modality’s predictions on non-target classes to guide those of the weaker modality, we provide the weak modality with richer semantic cues — namely, the dominant modality’s understanding of non-target categories. This effectively enhances the weak modality’s comprehension of non-target classes and improves its overall performance.

W3 & W9 & W12 & W17: Multimodal learning & Late-fusion & Static and dynamic late fusion?

(1) Multimodal Learning (MML) refers to learning comprehensive information from multiple modalities and effectively integrating them [1-3], which can be mainly categorized into early, mid-level, and late fusion.

(2) Late fusion typically refers to combining the outputs of different models as the final prediction [4-5]. Formally, each modality $x^{(m)}$ is passed through a modality-specific projection function $f^{(m)}$, resulting in an output $z^{(m)} = f^{(m)}(x^{(m)}) \in \mathbb{R}^C$, where $C$ is the number of classes. These outputs are then fused as:

$z^{\text{final}} = \sum_{m=1}^M w^{(m)} \cdot z^{(m)}$

Here, $w^{(m)}$ is the fusion weight for the $m$-th modality, and $\sum_{m=1}^M w^{(m)} = 1$.

• Static late fusion uses fixed weights $w^{(m)}$ across all samples. It assumes the reliability of each modality is consistent.
• Dynamic late fusion allows $w^{(m)}$ to vary across samples, often based on the confidence or uncertainty of each modality’s output. This enables the model to adaptively rely more on the high-quality modality for each input.

W5: Increasing the probability of target class and suppressing the probabilities of non-target class.

Using the dominant modality's target class probability to guide that of the weaker modality essentially performs single-class guidance, which is similar to applying a soft-label constraint on the target class. This overlaps with the supervision already provided by the cross-entropy loss, thus offering limited benefit.

In contrast, constraining the non-target class predictions of the weaker modality using those from the dominant modality provides richer semantic information. This complements the label-based cross-entropy loss and effectively enhances the weaker modality’s understanding of non-target classes, thereby improving its overall performance.

It is worth noting that we do not use the full output distribution of the dominant modality to guide the weaker one, which would risk overriding the unique information captured by the weaker modality. Instead, we selectively guide only the non-target class predictions, which allows us to inject informative signals from the dominant modality while preserving the individuality of the weaker one.

W6: Right side of block diagram in Fig 1a.

The weak modality is aligned with the dominant modality across all classes. In contrast, the bottom part of Fig. 1(a) illustrates the dominant modality guiding the weak modality specifically on the non-target classes.

In our formulation, the specific information refers to the probability of the target class, while the common information refers to the probabilities of the non-target classes.

• Existing methods (top) tend to align both specific and common information simultaneously, which may interfere with the unique information of the weak modality.

• Our method (bottom) constrains only the non-target class predictions, thereby introducing guidance from the dominant modality while preserving the unique information of the weak modality.

W7: Video error rate.

The bar chart shows cases where the audio modality errs while video is correct. The line chart tracks how many previously correct video samples become incorrect in the next epoch (Video error rate). This demonstrates that aligning a modality (video) with another across all classes can lead to a over-alignment collapse.

W8: Does audio model also change?

The audio model changes during training , as both the video and audio models are trained simultaneously.

W10 & W15: Equations in Fig. 2 & Equation (5).

(1) In Figure 2, we illustrate the training losses (CE loss and MNL loss) and highlight MNL lifts the multimodal robustness lower bound during optimization.(W10)

(2) Eq. 5 presents the specific form of the MNL loss. It uses the RDM’s predictions to guide IM on non-target classes.

W13 & W14 : Decision space instability&uncertainty

Decision space instability refers to a model's high sensitivity to input perturbations near the decision boundary, where even small changes can lead to significant shifts in prediction. And uncertainty can be intuitively viewed as low confidence.

The weak modality itself is relatively unstable, which, according to Eq. (4), may reduce the overall robustness of the multimodal system after fusion. In contrast, MNL enhances the UCoM of the weak modality, thereby improving the overall robustness of the system.

W19: Results on additional datasets for Table 3 & 4.

We extended our approach and ablation studies to three modalities datasets, the CMU-MOSEI. The results are presented as follow:

W20: Details on hyperparameters.

We have provided some implementation details in Line 215–218 and Line 94-114 of Supp. C. For the wiegits of CE and MNL losses, the CE weight is fixed at 1 and the MNL weight is varied from 0.2 to 2.0 under different noise levels, we set weight to 1.

W21: $\epsilon$.

We followed the setting of $\epsilon$ in QMF [7] and TMC [8], and unified the noise levels to 0, 5, and 10 for clearer comparison.

For Gaussian noise, it is noise intensity; for salt-and-pepper noise, it is the probability of noise occurrence; for audio, it is the SNR-based noise level; and for text noise, it denotes the probability of blank noise.

W22: RDM and IM.

• Even if the RDM and IM can be segregated, this does not imply that the performance of the unimodal RDM surpasses that of the fusion of RDM and IM. This is evident from the unimodal versus multimodal fusion results shown in Tables 1 and 2.

• We utilize both RDM and IM to integrate complementary information. Extensive research has shown that multimodal fusion generally outperforms unimodal [6,7].

Q1: Robustness radius R(x_i).

There might be some misunderstanding here. Our model aims to increase the robustness radius $R(x)$, where a larger UCoM indicates a favorable phenomenon. Specifically, $R(x_i)$ is defined as the minimum $\ell_2$ perturbation required to flip the model’s prediction from the ground-truth class $y$ to any other class. A larger $R(x_i)$ signifies higher robustness to input perturbations.

According to Eq. 4, a higher UCoM corresponds to a larger $R(x)$, which means stronger robustness of the multimodal system. MNL leverages the modality with high UCoM to guide the modality with low UCoM, thereby improving the latter’s UCoM and ultimately enhancing the overall system robustness.

Presentation issues.

We have standardized the use of the terms unimodal and late fusion, corrected the spelling error in Fig. 1(b), and refined Theorem and notations. Finally, we have carefully reviewed and revised all representation issues throughout the paper.

[1] Multimodal machine learning: A survey and taxonomy. TPAMI2018

[2] Multimodal learning with transformers: A survey. TPAMI2023

[3] Multimodal deep learning. ICML2011

[4] PDF. ICML2024

[5] QMF. ICML2023.

[6] DUA-Nets. AAAI2021

[7] CRMT. ICLR2024

[8] OGM-GE. CVPR2022

[9] MMPareto. ICML2024