Improving Multimodal Learning Balance and Sufficiency through Data Remixing

Weakness1: There are no concrete measurements to demonstrate the efficiency of Remix.

Response:

• Our method focuses on variations at the data level and we have reported size of training set in Table 2 of the main text to prove the efficiency.
• We measure the training time of 4 methods under the same conditions as shown below. We observe that sample-level evaluations tend to increase the training time, whereas Remix does not expand the training set, making it more efficient.

$\quad$

Weakness2: Lack of justification for the effectiveness of the KL metric and evaluation of unimodal performance.

Response:

• We are inspired by the measurement of uncertainty in Active Learning when choosing KL divergence as a sample-level metric and select specific training samples.
• We have provided training results using other feasible metrics for comparison with results presented in the following table and summary their weaknesses below.
  • Entropy: Mathematically equivalent with KL.
  • Loss: Loss tends to overly bias the data toward the weak modality, preventing the strong modality from effectively training.
  • Shapley: Shapley values sometimes fail to distinguish between modalities, as noted in Footnote 1 of the main text.

• Another important point is that by observing the unimodal accuracy, we can find that Remix alleviates the insufficient modality learning, where the strong modalities are also improved.

$\quad$

Weakness3: The performance in other modality combinations and multimodal scenarios.

Response:

• Our method is not restricted to specific modalities. In our theoretical analysis, we make no prior assumptions about modality properties, ensuring its general applicability. Meanwhile, the key steps of Remix —decoupling and reassembling—are modality-agnostic. The process only considers the accuracy relationship between modality pairs at sample-level without imposing any constraints on the modality type.
• Our method is not limited by the number of modalities. As the number of modalities increases, our method remains applicable by simply retaining the modality with the lowest KL divergence during the decoupling process. The selection mechanism remains valid in more complex scenarios involving three or more modalities.
• To further demonstrate the broad effectiveness of Remix, we conduct additional experiments on the UCF101 with two modalities(optical flow, vision) and the CMU-MOSEI with three modalities(text, vision, audio). As shown in the table, Remix consistently improves performance, further validating its wide applicability.

$\quad$

Weakness4: About Dropout and Head in ablation study.

Response:

• Dropout and Head are strategies to obtain unimodal outputs. Dropout refers to masking other modalities and using the output of the multimodal classification head as the unimodal output. Head involves adding an independent classification head to each encoder and using its output as the unimodal output.
• In Remix method, we select the Head approach for more accurate unimodal results. To synchronously update the parameters, we incorporate a unimodal loss into the loss function, which has been proved to improve multimodal capabilities. This ablation study aims to demonstrate that the improvements brought by Remix do not stem from the introduction of unimodal loss.

$\quad$

Weakness5: How to ensure that using KL divergence doesn't cause the model to learn from excessively noisy signals?

Response:

• We also encountered similar issues in our experiments. Therefore, we have acknowledged this potential limitation in the summary of main text. Our current approach to address this challenge involves setting additional thresholds.
• In UCF101, we observe that the KL divergence for optical flow are consistently smaller than those for visual. We adopted two strategies:
  • Scaling-Based Adjustment: We introduce a hyperparameter $\beta$ to scale the KL divergence of the optical flow and compare $KL_{\text{video}}$ with $\beta \times KL_{\text{flow}}$, achieving a performance of 84.59%.
  • Minimum KL Threshold: We set a lower bound $\alpha$ on KL divergence to ensure the model does not overly focus on samples with minimal information content, resulting in a performance of 84.01%.

Weakness1: The applicability of the Remix method on three or even more modalities.

Response:

• Our method is not limited by the number of modalities. As the number of modalities increases, our method remains applicable by simply retaining the modality with the lowest KL divergence during the decoupling process. The selection mechanism remains valid and ensures that our method remains effective in more complex scenarios involving three or more modalities.
• To further demonstrate the broad effectiveness of Remix, we conduct additional experiments on CMU-MOSEI with three modalities(text, vision, audio). As shown in the table, Remix consistently improves performance, further validating its wide applicability.

$\quad$

Weakness2: Analysis of modality imbalance in fusion-based decision-making.

Response:

• We conducted a further analysis of feature fusion during model decision-making. We observe that modality imbalance is not limited to unimodal encoders but also manifests significantly at the feature fusion layer. Taking the concatenation method as an example, both the audio and video modality outputs are 512-dimensional, resulting in a 1024-dimensional fused feature. We compute the weighting distribution of each modality, and the results are presented in the table below.

• Our findings indicate that the Remix method not only promotes modality balance at the unimodal encoder level but also enhances balance at the fusion layer.

• For more complex models(MMTM or CentalNet), the improvements brought by Remix are relatively smaller. We attribute this to the fact that early-stage interactions may be somewhat suppressed due to our decoupling process. However, this trade-off is offset by the enhanced unimodal capabilities, which ultimately contribute to overall performance improvements.

$\quad$

Weakness3: Other metrics to evaluate the performance of the modalities to determine the decoupling results?

Response:

• We are inspired by the measurement of uncertainty in Active Learning when choosing KL divergence as a sample-level evaluation method and a criterion for selecting modality-specific training samples.
• We have provided training results using other feasible metrics for comparison with results presented in the following table and summary their weaknesses below.
  • Entropy: In classification tasks, Entropy and KL divergence are mathematically equivalent.
  • Loss: Loss tends to overly bias the data toward the weak modality, causing preventing the strong modality from effective training.
  • Shapley Value: Shapley values sometimes fail to distinguish between modalities, as noted in Footnote 1 of the main text.

Weakness1: Modality Clash and Modality Imbalance

Response:

• In summary, Modality Imbalance is unidirectional, while Modality Clash is bidirectional. Modality Imbalance refers to a scenario where a strong modality dictates the learning process, preventing other modalities from being sufficiently trained. Modality Clash describes interference between modalities. Even if modality balance is achieved, differences between modalities may still lead to insufficient learning across all modalities.
• The accuarcy of unimodal capabilities presented in the following table demonstrate that all modalities benefit from our approach, further validating its effectiveness.

$\quad$

Weakness2: The Mutual Information in Multimodal Tasks.

Response: We understand the reviewer's concerns regarding the potential decrease in MI due to alternating unimodal training. This issue is carefully considered in our approach.

• First and foremost, it is important to clarify that our task is multimodal co-decision, which involves integrating multiple modalities to make a final decision—abstractly represented as A + B → C. The goal is to perform cross-modal feature selection and integration based on the learning outcomes of individual modalities. This differs from tasks such as retrieval, where decision-making relies on capturing shared information across modalities—abstractly represented as A → B. These tasks require highlighting consistency information through similarity measurements, which inherently mitigates modality imbalance issues present in co-decision tasks.
• Following the reasons, existing co-decision network architectures are inherently weakly interactive. For example, they often rely on direct concatenation of unimodal representations or gated fusion at the decision level. As a result, our data decomposition does not significantly impact interaction, since these models already operate with minimal cross-modal dependency. By conducting experiments measuring the MI between modality-specific features before and after applying our method, we prove our perspective.

• Therefore, in co-decision tasks, the performance bottleneck lies in modality imbalance and modality clash, which hinder effective unimodal learning and limit cross-modal synergy. Our work is specifically designed to address this issue by ensuring that each modality is adequately learned before integration, ultimately improving overall model performance.
• Additionally, we provide extra data to further support our argument. We construct a confusion matrix analyzing the relationship between model decision correctness and the presence of correctly predicted modalities. The results can be found at: https://anonymous.4open.science/r/ICM-Rebuttal-17C8/Fig1.png. By analyzing it, we find that in co-decision tasks, it is common for a model to have at least one modality predict correctly while the final decision is incorrect. However, after applying the Remix method, this type of misjudgment is significantly reduced. This suggests that Remix can guides samples to the appropriate modality space, and improves decision-making accuracy.

$\quad$

Weakness3: The method and experiments are limited.

Response:

• Our method is not restricted to specific modalities. In our theoretical analysis, we make no prior assumptions about modality properties, ensuring its general applicability. Meanwhile, the key steps of Remix —decoupling and reassembling—are modality-agnostic. The process only considers the accuracy relationship between modality pairs at sample-level without imposing any constraints on the modality type.

• Our method is not limited by the number of modalities. As the number of modalities increases, our method remains applicable by simply retaining the modality with the lowest KL divergence during the decoupling process. The selection mechanism remains valid and ensures that our method remains effective in more complex scenarios involving three or more modalities.

• To further demonstrate the broad effectiveness of Remix, we conduct additional experiments on the UCF101 with two modalities(optical flow, vision) and the CMU-MOSEI with three modalities(text, vision, audio). As shown in the table, Remix consistently improves performance, further validating its wide applicability.

  	Baseline	GBlend	OGM	PMR	Reample	MLA	Remix
  UCF101	80.78	82.82	82.55	81.87	84.09	83.03	84.59
  CMU-MOSEI	83.32	84.45	85.03	84.13	84.50	82.84	85.89

Weakness1: The method and experiments are limited to certain modalities.

Response:

• Our method is not restricted to specific modalities. In our theoretical analysis, we make no prior assumptions about modality properties, ensuring its general applicability. Meanwhile, the key steps of Remix —decoupling and reassembling—are modality-agnostic. The process only considers the accuracy relationship between modality pairs at sample-level without imposing any constraints on the modality type.

• Our method is not limited by the number of modalities. As the number of modalities increases, our method remains applicable by simply retaining the modality with the lowest KL divergence during the decoupling process. The selection mechanism remains valid and ensures that our method remains effective in more complex scenarios involving three or more modalities.

• To further demonstrate the broad effectiveness of Remix, we conduct additional experiments on the UCF101 with two modalities(optical flow, vision) and the CMU-MOSEI with three modalities(text, vision, audio). As shown in the table, Remix consistently improves performance, further validating its wide applicability.

  	Baseline	GBlend	OGM	PMR	Reample	MLA	Remix
  UCF101	80.78	82.82	82.55	81.87	84.09	83.03	84.59
  CMU-MOSEI	83.32	84.45	85.03	84.13	84.50	82.84	85.89

$\quad$

Weakness2: Effectiveness on Text modality and MLLMs.

Response:

• Multimodal Large Language Models (MLLMs) differ significantly from our task in terms of architecture and underlying principles. MLLMs typically use the language modality as the foundation, mapping other modalities onto it. This differs from the modality imbalance issue we address in co-decision tasks. That's also the main difference between our method and ImageBind.
• Additionally, we follow the existing model structures in the Balanced Multimodal Learning (BML) domain to ensure a fair comparison and demonstrate the effectiveness of our proposed method.
• Additionally, we have demonstrated the effectiveness of the text modality within our method. In table above, we have supplement our results with the CMU-MOSEI dataset, which includes text, video, and audio modalities. In the baseline model, the accuracy of the text modality was 79.96%, and after applying Remix, it improved to 81.29%. This result indicates that Remix is also beneficial for the language modality, further validating its effectiveness across different modality types.

$\quad$

Weakness3: There are no theoretical claims.

Response:

• In this paper, we introduce a novel phenomenon termed modality clash, which defines the bidirectional interference between modalities in multimodal learning (as illustrated in Figure 3 in main text, we quantify the optimization direction deviation of strong modalities). This perspective differs from the traditional modality imbalance problem, which primarily emphasizes the one-way suppression of weaker modalities by stronger ones.
• Accordingly, our Remix method is designed to directly address this issue. To mitigate modality clash, we need to control the composition of each training batch, which is achieved through the assembling step. To select appropriate samples and enable batch control, we first perform decoupling of multimodal inputs based on sample-level evaluation.