Adaptive Re-calibration Learning for Balanced Multimodal Intention Recognition

We are deeply grateful for your insightful and positive review of our work. We sincerely appreciate your recognition of our paper's strengths, including the well-motivated and effective solution to modality imbalance, the technical novelty and computational efficiency of the Leave-One-Out (LOO) valuation method, and the strong empirical validation.

We also thank you for your constructive feedback on the areas for improvement. Your suggestions regarding the formal definition of modality imbalance and the clarity of the training procedure are highly valuable. We agree that addressing these points will significantly strengthen the paper's theoretical grounding and reproducibility. Below, we address your questions in detail.

---

Regarding Weakness 1: Can you provide a formal definition and measurement of “modality imbalance”?

Thank you for this excellent suggestion. We agree that a formal definition and a quantitative metric for modality imbalance would greatly enhance the rigor of our paper. We will incorporate this into the revised manuscript.

1. Formal Definition: We will formally define modality imbalance as: "The degree of non-uniformity in the contributions of different modalities to the final prediction for a given multimodal sample."

2. Quantitative Metric (Modality Imbalance Index): As you astutely suggested, we can leverage our Leave-One-Out (LOO) contribution scores to create a quantitative metric. For a sample $i$ with $M$ modalities, we have a vector of contribution scores $\Delta_i = \{\delta_i^{(m_1)}, \delta_i^{(m_2)}, ..., \delta_i^{(m_M)}\}$, where $\sum_{m} \delta_i^{(m)} = 1$. An ideal balance implies that all modalities contribute equally (i.e., $\delta_i^{(m)} = 1/M$ for all $m$).

To quantify the deviation from this ideal state, we will introduce the Modality Imbalance Index (MII), defined as the variance of these contribution scores:

$\text{MII}(i) = \text{Var}(\Delta_i) = \frac{1}{M}\sum_{m=1}^{M} \left(\delta_i^{(m)} - \frac{1}{M}\right)^2$

A higher MII value indicates a greater imbalance for sample $i$, signifying that one or a few modalities are dominating the prediction.

3. Connection to ARL: We will further clarify that our proposed ARL framework is designed to implicitly minimize this imbalance.

• Contribution-Inverse Sample Calibration (CISC) acts as a reactive, sample-level mechanism. It directly targets samples with high MII by masking dominant modalities whose contribution $\delta_i^{(m)}$ exceeds a threshold $\tau$, thereby encouraging the use of underutilized modalities.
• Weighted Encoder Calibration (WEC) serves as a proactive, architectural-level mechanism. It adjusts encoder weights based on global contribution statistics ($\bar{\delta}^{(m)}$) and stability, aiming to reduce the average modality imbalance across the entire dataset over the long term.

We will add this formal definition and the MII metric to Section 3.1 (Problem Formulation) in our revised paper. We believe this will provide a clearer theoretical foundation for the problem we are solving.

---

Regarding Weakness 2: Could you provide a pseudocode or flowchart of the full ARL training process?

We thank you for pointing out the need for a clearer description of our training procedure. We agree that a step-by-step algorithm is essential for understanding and reproducibility. In the revised manuscript, we will add a detailed pseudocode of the ARL training process. Below is a draft of the algorithm we plan to include.

Algorithm 1: Adaptive Re-calibration Learning (ARL) Training Procedure

---

Require: Training set $\mathcal{D}_{train}$, Validation set $\mathcal{D}_{val}$, Base model $f_{\theta}$
Require: ARL hyperparameters: $\tau, T, \lambda, \alpha$

Initialize model parameters $\theta$

for epoch = 1 to max_epochs do
    for each batch in $\mathcal{D}_{train}$ do
        // Calculate sample-wise modality contribution (Sec. 3.2)
        Compute full-modality features $F_i$ and masked features $F_i^{(-m)}$ for each sample.
        Calculate contribution scores $\delta_i^{(m)}$ from the feature similarities.

        // Calibrate features with CISC (Sec. 3.3)
        Calibrate features $\tilde{x}_i^{(m)}$ by masking if a sample's contribution $\delta_i^{(m)} \ge \tau$.

        // Optimize model using calibrated features
        Compute final logits from the calibrated features $\tilde{x}_i$.
        Calculate prediction loss $\mathcal{L}$.
        Update model parameters $\theta$ via backpropagation.
    end for

    // Periodically calibrate encoders with WEC (Sec. 3.4)
    if epoch % T == 0 then
        Calculate stability score $\mathcal{P}_m$ on $\mathcal{D}_{train}$ and $\mathcal{D}_{val}$.
        Calculate average contribution score $\bar{\delta}^{(m)}$ on $\mathcal{D}_{val}$.
        Compute dynamic weights $w_m$ from $\mathcal{P}_m$ and $\bar{\delta}^{(m)}$.
        Adjust encoder weights $\theta^{(m)}$ using the computed weights $w_m$.
    end if
end for

---

We will place this algorithm in the Methodology section of our revised paper, accompanied by explanatory text. We are confident this will fully clarify the operational flow of ARL and its integration with base models.

Once again, we thank you for your time and constructive suggestions. We believe these revisions will make our paper stronger, clearer, and more impactful.

Dear Reviewer G2eC,

We would like to extend our sincerest gratitude for your time and insightful feedback on our manuscript. Your comments are invaluable and have helped us identify areas for significant improvement. We have carefully considered each of your points and provide our responses below. We hope our clarifications and additional results will address your concerns.

---

Regarding Weakness 1: Comparison with existing imbalanced learning methods.

We thank the reviewer for this valuable suggestion. We would like to begin by clarifying our experimental motivation. As reviewer foUd noted, ARL is designed as a "plug-and-play" module to enhance existing SOTA models. Our primary focus is on addressing modality imbalance within the specific domains of intention and sentiment analysis. Therefore, our initial step was to demonstrate ARL's ability to consistently improve strong, representative baselines from this domain (i.e., MAG-BERT, MulT, TCL-MAP), as shown in Tables 1 and 2 of our paper.

To further address your concern directly, we have now conducted new experiments comparing ARL with several representative imbalanced learning methods by applying them to the MAG-BERT baseline on the MIntRec dataset. The results are presented below. We will add this table and corresponding analysis to the appendix in our revised manuscript.

As shown, our proposed ARL outperforms these strong baselines for imbalanced learning, which further validates its effectiveness.

[1] P. Morerio, et al, “Curriculum dropout,” in ICCV 2017

[2] X. Peng, Y. Wei, et al, “Balanced multimodal learning via on-the-fly gradient modulation,” in CVPR 2022

[3] Y. Wei, et al. "On-the-fly modulation for balanced multimodal learning." IEEE TPAMI 2024

[4] C. Huang, Y. Wei, et al. "Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisition" in CVPR 2025

---

Regarding Weakness 2: Efficiency of the LOO method.

We appreciate your concern regarding the computational cost of our Leave-One-Out (LOO) modality valuation method. However, we believe there may have been a misunderstanding of LOO's core contribution to efficiency.

The primary challenge in sample-level modality valuation lies in the exponential complexity ($O(n2^n)$) of theoretically-grounded methods like Shapley values, which renders them impractical for real-world MIR systems with multiple modalities. The central contribution of our LOO method is precisely in overcoming this critical bottleneck by reducing the complexity from exponential to linear ($O(n)$).

We have provided a detailed analysis of this in Section 4.2 and visualized it in Figure 3 of our paper. These results clearly demonstrate that LOO is dramatically more scalable and efficient than the Shapley-based approaches. Therefore, achieving linear complexity is not a weakness but a key strength of our work, making dynamic, sample-specific modality calibration computationally feasible for the first time in this context.

---

Regarding Weakness 3: Insufficient analysis of hyperparameters.

Thank you for pointing out the need for a more thorough hyperparameter analysis. We did perform extensive tuning during our experiments, and we agree that including these results strengthens the paper. Due to space constraints in the initial submission, this analysis was omitted.

To address your comment, we now provide the sensitivity analysis for the masking threshold $\tau$ using the MulT model on MIntRec. We will include this in the appendix of the revised manuscript.

Optimal performance is achieved at $\tau = 0.5$, and we observed similar robust trends across our other experimental setups. This analysis demonstrates that while the hyperparameter is important, its optimal value can be robustly identified via standard tuning procedures.

In our revised version, we will add an appendix containing a detailed justification for our per-dataset tuning strategy, along with the complete sensitivity analysis tables for these parameters.

---

Regarding Weakness 4: Lack of experimental evidence for sample-level calibration.

We sincerely thank the reviewer for this insightful and constructive comment. We agree that providing direct experimental evidence to demonstrate ARL's sample-level calibration mechanism is crucial and will significantly strengthen our paper. The reviewer’s point is well-taken; while our method is designed to be sample-centric, our original experiments only showed its aggregated effects on the entire test set.

We would also like to respectfully note that our sample-level focus aligns with a significant research direction in the community. For instance, the recent work by Wei et al. [5] (CVPR 2024) also emphasizes valuing modality contributions on a per-sample basis, underscoring the growing recognition that modality importance is context-dependent rather than static.

[5] Wei et al., "Enhancing multimodal cooperation via sample-level modality valuation" (CVPR 2024)

Therefore, the reviewer’s suggestion to provide more direct, sample-centric evidence is particularly relevant. To that end, we have prepared the following quantitative analysis, which we will add to the revised manuscript to compellingly illustrate the effectiveness of ARL's sample-level adaptation.

Quantitative Grouped Analysis: Statistical Validation

To provide broad statistical evidence, we partitioned the MIntRec test set based on a Sample Imbalance Score, which we defined as the variance of the three LOO modality contribution scores for each sample. A higher score signifies greater imbalance. We then evaluated the performance of the baseline MAG-BERT and our ARL-enhanced model on these groups.

As the results clearly show, the performance gain from ARL is substantially larger (+9.5%) on samples with high modality imbalance. This quantitatively demonstrates that our training-time calibration strategy produces a final model that is specifically more robust to the challenging sample-level imbalances it was designed to address.

We are confident that these new results provide the direct evidence requested by the reviewer. We thank the reviewer again for pushing us to make this important addition to our work.

---

Once again, we thank you for your rigorous review. We believe that with the proposed clarifications and additional experiments, our manuscript has been substantially strengthened. We hope that our responses have adequately addressed your concerns.

Dear Reviewer A7Ss,

We are sincerely grateful for your time and for providing such insightful and constructive feedback on our manuscript. Your questions have allowed us to clarify key aspects of our methodology and further strengthen our paper. We are pleased to address each of your points below.

---

Regarding Weakness 1: The Consideration of Soft Masking.

We thank you for this excellent question. Your suggestion prompted us to conduct a direct comparative experiment to validate our design choice of using hard masking in the Contribution-Inverse Sample Calibration (CISC) component.

Our initial motivation for hard masking was to introduce a strong regularization effect to decisively break the model's learned over-reliance on dominant modalities. To empirically test this against a less aggressive alternative, we implemented a soft masking variant and evaluated both approaches with the MAG-BERT model on the MIntRec dataset. The results are as follows:

The results clearly show that our proposed hard masking approach yields significant and consistent improvements across all evaluation metrics. While soft masking provided a modest improvement in Accuracy and Precision, it was detrimental to the F1-score and Recall.

We hypothesize that this is because soft masking, by only attenuating a dominant modality's features, is an insufficient signal to compel the model to fundamentally alter its strategy. Hard masking, in contrast, creates a necessary "information vacuum" by completely ablating the over-relied-on features for that instance. This forces the model to actively learn from underutilized modalities, leading to a more robust and balanced representation, as reflected in the superior F1 and Recall scores.

Action: We will add this comparative experiment and analysis to the Ablation Studies section in our revised manuscript. This addition, prompted by your valuable feedback, provides a strong empirical justification for our design choice.

---

Regarding Weakness 2: The modality Independence.

Thank you for raising this important theoretical point. We wish to clarify that our use of the LOO method does not assume modality independence. Rather, it serves as a computationally efficient, empirical tool to approximate the marginal contribution of a modality to the final, fused representation in the specific context of all other available modalities.

The method inherently accounts for inter-modality dependencies because the contribution is measured by the change in the fused representation upon a modality's removal.

• If two modalities are redundant, removing one will cause only a small change in the fused output, as the other modality compensates. LOO correctly assigns a low marginal contribution score in this case.
• If two modalities are complementary, removing one will cause a significant change, correctly resulting in a high contribution score.

Thus, our LOO valuation implicitly captures the effects of inter-modality dependencies by operating on the joint, fused output.

Action: To make this explicit, we will add a clarification in Section 3.2 of our revised manuscript, stating that the LOO method empirically assesses marginal contributions within the context of modality fusion, thereby accounting for their interactions.

---

Regarding Weakness 3. & 4: The Tuning of Hyperparameters $\tau$ (Eq. 10) and $\eta$ (Eq. 9).

Thank you for these questions regarding our experimental methodology. We agree that transparently detailing our hyperparameter tuning is essential for reproducibility.

Both $\tau$ and $\eta$ are tuned as a single scalar value per dataset, determined via grid search on the corresponding validation set. We would like to elaborate on why we chose this over a per-modality approach. In our preliminary designs, we explored per-modality thresholds but found this to be less effective. This is because, for a given dataset, one modality is often consistently dominant. The threshold's primary role is to govern the masking rate of this specific modality. Applying a separate threshold to an already weak modality has a negligible impact, as it rarely becomes dominant enough to be masked, yet it significantly increases tuning complexity. A single, data-driven threshold per dataset proved to be the most effective and practical solution.

For example, the following table shows the sensitivity analysis for $\tau$ with the MulT model on the MIntRec validation set:

Optimal performance is achieved at $\tau = 0.5$, and we observed similar robust trends across our other experimental setups. The same rationale and tuning procedure apply to the parameter $\eta$.

Action: In our revised version, we will add an appendix containing a detailed justification for our per-dataset tuning strategy, along with the complete sensitivity analysis tables for both $\tau$ and $\eta$ for each dataset.

---

Once again, we thank you for your thoughtful and rigorous review. We hope that our responses and planned revisions have fully addressed your concerns. We believe the resulting manuscript will be significantly improved.

To reviewer foUd:

Thank you for your detailed and valuable feedback. We are pleased to report that we have conducted new experiments and analyses based on your suggestions, and we believe the results significantly strengthen our paper. Below are our detailed responses.

---

Regarding Weakness 1: Overall Running Efficiency and Practicality

We appreciate your concern regarding the computational cost. To provide a clear and fair assessment, we have benchmarked ARL's training and inference time against the baseline, as well as against other state-of-the-art modality balancing methods, including the Shapley value-based approach [1] and InfoReg [2].

The results, summarized in the tables below, demonstrate that ARL offers a highly competitive balance of efficiency and performance.

Table A: Efficiency and Performance on MOSI (3 modalities)

Table B: Efficiency and Performance on MOSI (2 modalities: {a, v})

Our key takeaways are:

• Reasonable and Competitive Overhead: ARL's training overhead is moderate and its total convergence time is highly competitive with (and even slightly faster than) other SOTA methods like InfoReg [2].
• Superior Efficiency to Shapley-based Methods: As theoretically motivated, ARL is significantly more efficient than Shapley value-based methods [1], confirming the practical advantage of our LOO approach.
• Zero Inference Cost: Crucially, ARL introduces no additional overhead during inference, making it highly practical for real-world deployment.
• Robust Performance Gains: In both three-modality and two-modality settings, the modest training cost translates into consistent and superior performance gains, highlighting a favorable trade-off between efficiency and robustness.

We will add this detailed analysis to the revised paper to assure readers of ARL's practicality.

[1] Y. Wei, et al. "Enhancing multimodal cooperation via sample-level modality valuation." in CVPR 24.

[2] C. Huang, Y. Wei, et al. "Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisition" in CVPR 2025

---

Regarding Weakness 2: State-of-the-Art (SOTA) Comparisons

Thank you for encouraging us to broaden our SOTA comparison. We have followed your suggestion and included results from recent SOTA methods [3, 4, 5].

Furthermore, to unequivocally demonstrate the generalizability and utility of our framework, we went a step further and applied ARL to some of these SOTA models. The results on the MOSI dataset are highly compelling:

Table C: ARL Boosts Performance of SOTA Models on MOSI

As shown, ARL consistently improves the performance of strong SOTA models like DMD and EMOE. This powerfully demonstrates the core value of our work: ARL is not just another competing model, but a versatile and orthogonal plug-and-play module that effectively addresses the fundamental problem of modality imbalance, thereby empowering a wide range of multimodal architectures. We will incorporate this table and discussion into our main paper.

[3] Contextual Augmented Global Contrast for Multimodal Intent Recognition. CVPR24.

[4] Decoupled multimodal distilling for emotion recognition. CVPR23.

[5] EMOE: Modality-Specific Enhanced Dynamic Emotion Experts. CVPR25.

---

Regarding Weakness 3 & 4: Missing References and Minor Issues

We are grateful for the suggested references and your careful proofreading. We will add discussions of the suggested papers to our Related Work section and have corrected the typos.

To further strengthen our paper and provide a more comprehensive comparison against the broader field of imbalanced learning (a point also raised by Reviewer G2eC), we have conducted new experiments comparing ARL with several representative imbalanced learning methods by applying them to the MAG-BERT baseline on the MIntRec dataset. The results are presented below. We will add this table and corresponding analysis to the appendix in our revised manuscript.

As shown, our proposed ARL outperforms these strong baselines for imbalanced learning, which further validates its effectiveness.

[6] P. Morerio, et al, “Curriculum dropout,” in ICCV 2017

[7] X. Peng, Y. Wei, et al, “Balanced multimodal learning via on-the-fly gradient modulation,” in CVPR 2022

[8] Y. Wei, et al. "On-the-fly modulation for balanced multimodal learning." IEEE TPAMI 2024

[9] C. Huang, Y. Wei, et al. "Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisition" in CVPR 2025

---

In conclusion, we have strived to addres your concerns with new experiments and detailed analysis. We believe these revisions have substantially strengthened the paper. Thank you once again for your constructive guidance.