Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportion

Thank you for your conductive comments.

Answer for Question "Lemma 1 Statement": Thank you for pointing this out. This was indeed a typographical error on our part. The corrected statement of the lemma is as follows:
Lemma 1 (Gap Bound) There exists a constant $\kappa$ such that the loss gap function $\mathcal{G}(\cdot)$ and the gradient norm of the strong modality satisfy the following relationship:

$||\nabla\mathcal{L}^a(\Phi_t)||\ge\kappa|\mathcal{G}(\Phi_t)|.$

We will address this issue in the final version.

Answer for Question: "Computational Cost vs The Number of Classifiers": The computational overhead—including both training and inference time—is indeed influenced by the number of classifiers. Since our method dynamically adds classifiers during training, we incorporate a threshold to limit the maximum number of classifiers. This allows us to systematically investigate the trade-off between computational cost and model performance as a function of classifier quantity.

Specifically, we conduct experiments on the CREMA-D dataset, where the maximum number of weak modality classifiers is constrained to not exceed a predefined threshold $M$. We vary $M$ across the set {2, 4, 6, 8, 10}, and for each setting, we report the training time, inference time, and the corresponding classification performance.

The experimental results are summarized in the following Table. We observe that as the number of classifiers increases, both training time and inference time increase accordingly, while performance also improves to a certain extent. This indicates that incorporating more classifiers can indeed enhance model performance, but at the cost of increased computational overhead.

Table 1. Computational cost vs the number of classifiers.

Relevant experiments and discussion will be included in the final version.

Answer for Question: "Ablation Study on Other Dataset": To further demonstrate the effectiveness of the key components of our method, we conduct the experiments for ablation study on trimodal dataset, i.e., NVGeasture dataset. Specifically, we analyze the effectiveness of the losses $\epsilon$, $\epsilon_o$, and $\epsilon_p$ respectively defined in Eq.(2), (3), and (4) of the original paper.

For video-audio, image-text, and tri-modal scenarios, we selected three representative datasets for ablation experiments: the video-audio dataset CREMA-D, the image-text dataset Twitter2015, and the tri-modal dataset NVGesture. The result on CREMA-D dataset is presented in Table 2 of the original paper. And the results on Twitter2015 and NVGeasture dataset are respectively shown in Table 2 and Table 3. We can find that both objectives in Eq. (2), (3), and (4) can boost the multimodal classification performance in terms of accuracy.

Table 2. Ablation study on Twitter2015 dataset.

Table 3. Ablation study on NVGeasture dataset.

Relevant experiments and discussion will be included in the final version.

Thank you for your conductive comments.

Answer for Question "Comparison with ReconBoost": Although both our method and ReconBoost [1] are inspired by the idea of gradient boosting, they differ fundamentally in motivation, implementation, and theoretical grounding:

• The two methods are driven by distinct objectives. Our approach aims to improve the classification performance of weak modality classifiers using gradient boosting, whereas ReconBoost adopts the concept of gradient boosting to mitigate modality competition.
• The utilization of gradient boosting differs significantly between these two methods. Our method learns the residual of a single-modality classifier to refine weak modality performance, while ReconBoost leverages the residual from another modality to address inter-modal competition.
• The theoretical foundations of the two methods diverge. Our method establishes a link between the gradient boosting algorithm and a performance gap function, whereas ReconBoost demonstrates that, under certain conditions, introducing KL divergence as a regularization term results in a training objective equivalent to a specific form of gradient boosting.

In summary, despite both methods drawing inspiration from gradient boosting, they embody fundamentally different motivations, mechanisms, and theoretical interpretations.

Relevant discussion will be included in the final version.

Answer for Question "Selection of Score Function": In our method, the score function is employed to quantify the disparity in classification performance between different modalities, which serves as the basis for determining whether to introduce an additional classifier for the weak modality. Therefore, any metric that effectively captures the difference in classification capabilities across modality-specific models can be adopted as a score function.

To some extent, entropy and loss can reflect the learning status of modality-specific models and thereby serve as proxies for their classification capabilities. To evaluate the effectiveness of using entropy and loss as score functions, we conducted experiments on the CREMA-D dataset, where these two metrics were used to guide the adaptive classifier assignment. The experimental results are presented in Table 1, where $\tau$ denotes the threshold. Please note that since we use the ratio between different modal metrics to compare against the threshold $\tau$, the value of $\tau$ remains consistent across different metric types such as confidence, loss, and entropy. These findings indicate that the proposed method is robust to the choice of score function, consistently achieving comparable results regardless of whether confidence score, entropy, or loss is used.

Table 1. Performance with different score function.

Relevant experiments and discussion will be included in the final version.

Answer for Question "Trimodal Experimental Details": For the scenario with three modalities, a key aspect of the experiment lies in determining whether to add an additional classifier based on the confidence scores of the three modalities. Specifically, we compare the confidence score of each modality with the average of the confidence scores of the other two. If the difference exceeds a predefined threshold, a new classifier is assigned to the corresponding modality.

Relevant implementation details will be included in the final version.

Reference:
[1]. Hua, Cong, et al. ReconBoost: Boosting can achieve modality reconcilement. ICML. 2024.

Thank you for your conductive comments.

Answer for Question "Adaptability for Other Boosting Strategy": First, our algorithm is inspired by the gradient boosting paradigm. The motivation behind this approach lies in improving the classification performance of weak modalities during training, for which residual learning provides a natural solution. Second, other representative methods such as AdaBoost also enhance classifier performance by modifying the sampling strategy, which is another feasible direction. We believe that resampling misclassified samples from the weak modality can facilitate more effective learning, potentially bridging our motivation with AdaBoost-style algorithms.

It is worth noting that certain existing works have investigated sample-level imbalance. For example, SMV [1] achieves balanced learning by resampling modality-specific data that has not been sufficiently learned, and has demonstrated promising results. However, integrating this sample-level resampling strategy into the weak modality boosting framework may constitute a novel research direction. As such, we leave this exploration for future work.

Relevant discussion will be included in the final version.

Answer for Question "Effectiveness with Early Fusion Architectures": Our method is built upon a late fusion architecture, with the core motivation of identifying and mitigating performance disparities across modalities, thereby achieving balanced classification capabilities among them. This implies that our method cannot be directly applied to architectures that rely solely on a single fusion classifier. In early fusion methods, only fused representations are used for prediction, and the classification process is not decomposable into modality-specific classifiers. As a result, our approach is not directly compatible with early fusion architectures.

To adapt our method to early fusion settings, it may be necessary to explicitly decouple the classification contributions of individual modalities and assess their performance differences. This would likely require a fundamentally new architectural design, which we consider a promising direction for future work.

Answer for Question "Explation of Full Prediction vs $t-1$ CLS": Our method adaptively adds new classifiers during the training process, and we denote the current number of classifiers for the weak modality as $t$. "Prediction of $t-1$ CLS" refers to the performance of the first $t-1$ classifiers, excluding the newly added one, while "Full Prediction" refers to the performance of all t classifiers.

The primary objective of this experiment is to verify the feasibility and necessity of residual learning by comparing the performance of the $t$-th classifier with that of the first $t–1$ classifiers. Essentially, the $t$-th classifier learns the residual—that is, the performance difference between itself and the previous classifiers. This observation is also supported by the results shown in Figure 3 of the original paper.

Answer for Question "Implementations of Toy Experiments": The toy experiment is designed to validate the motivation behind our method. Specifically, it is conducted on the CREMA-D dataset, where the task is multimodal classification. Four methods are adopted for comparison: Naive MML, G-Blend [2], MML w/ GB, and Ours. Here, MML w/ GB refers to applying the gradient boosting algorithm to further improve the trained video model obtained from Naive MML, while keeping the audio model fixed. Ours refers to the method employing the sustained boosting algorithm proposed in this paper. Furthermore, ResNet-18 is used as the encoder for all methods, with its parameters randomly initialized. The selection of other hyperparameters is kept consistent with the main experiment.

Relevant implementation details will be included in the final version.

Reference:
[1]. Wei, Yake, et al. Enhancing multimodal cooperation via sample-level modality valuation. CVPR. 2024.
[2]. Wang, Weiyao, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard?. CVPR. 2020.

Thank you for your conductive comments.

Answer for Question "Influence of Unstable Confidence Score": Since our method relies on confidence gap to determine whether to add a classifier, fluctuations in confidence scores are indeed an influential factor. We conduct an experiment to explore the fluctuation of confidence score in scenario with modality noise on CREMA-D dataset. Specifically, we add Gaussian noise to the video modality to introduce instability in confidence scores and investigate its impact on the algorithm. We design a metric to evaluate the fluctuation of confidence score during training. We first compute the absolute difference in confidence scores between two consecutive iterations, and then take the average over all training rounds, i.e., $\bar{s}=\frac{1}{n}\sum_{t=2}^n|s_t-s_{t-1}|$. Here, $s_t$ denotes the confidence score at $t$-th iteration. A higher value of $\bar{s}$ indicates more severe fluctuations in confidence scores, reflecting a sharper and less stable prediction behavior. Furthermore, we compare our method with naive MML and ReconBoost [1].

We first compare the model performance in scenario with modality noise in Table 1, where $\mu$ denotes the noise rate [2], #CLS denotes the number of classifier during training, and in the first column, we also report the fluctuation of confidence score. In fact, as training progressed, we observe that in noisy scenarios, increasing the number of classifiers beyond a certain point no longer led to improved performance on the validation set. This phenomenon became more pronounced as the noise level increased. In other words, the classifiers may overcompensate in scenarios involving modality noise. On the other hand, we also observe that after reaching its optimal performance, the model's performance gradually degraded without a drastic drop. Therefore, in practical scenarios, an early stopping strategy based on validation set performance can be employed to select the optimal model and avoid classifier overcompensation.

Table 1. The fluctuation of confidence score.

Furthermore, we compare the performance with competitive baselines including naive MML, ReconBoost [1]. The results are shown in Table 2. We can find that compared with the competitive method ReconBoost, our approach consistently achieves superior performance, demonstrating its effectiveness in scenario with modality noise.

Table 2. Performance comparison under modality noise scenario.

Relevant experiments and discussion will be included in the final version.

Answer for Question "Stability under Modality Missing": Our method is adaptable to scenarios involving missing modalities. We comprehensively evaluate its performance under test-time missing modality conditions. Specifically, test-time missing refers to cases where the modalities are complete during training but missing during the testing phase. The experiments are conducted on CREMA-D dataset with different missing rate [3]. Naive MML, ReconBoost [1], and MLA [3] are selected as baselines for comparison, where MLA introduces specifically designed algorithms to address the corresponding challenges in modality missing. We report the results with missing rate 20% and 50% in Table 3. We can find that as the modality missing rate increases, the performance of all methods declines. Nevertheless, our method consistently achieves the best performance under all missing rates, demonstrating the effectiveness of our method in scenario with missing modality.

Table 3. Performance comparison in scenario with modality missing.

Relevant experiment and discussion will be included in the final version.

Answer for Question "Modality-Specific Knowledge Distillation Strategy": Using strong modalities to distill weaker ones is indeed a compelling and insightful idea. In our approach, we utilize performance gap to determine the relative strength of each modality and then independently enhance the classification capability of the weaker modalities. This can be viewed as an implicit strategy that leverages modality performance to boost the learning of weak modalities. In contrast, methods that explicitly distill knowledge from strong modalities aim to directly improve the classification performance of weaker ones through knowledge transfer. For instance, C2KD [4] is a representative method designed based on this idea by using the confidence scores from the strong modality to guide and refining the confidence of the weak modality. We conducted experiments to compare the effectiveness of our method and C2KD.

We compare this approach with our method on CREMA-D dataset under the same setting. The accuracy results are reported in Table 5, where the accuracy of C2KD is refered from its original paper. We can find that (1). Compared with naive MML, C2KD can achieve better performance, demonstrating the effectiveness of modality-specific knowledge distillation; (2). Our approach can achieve the best performance compared with naive MML and C2KD. This suggests that directly enhancing the performance of weak modality classifiers may be a more effective strategy than knowledge distillation.

Table 5. The effectiveness of cross-modal knowledge distillation.

Relevant experiments and discussion will be included in the final version.

Answer for Question "Computational and Memory Cost": To demonstrate the practical applicability of our approach, we further provide a comprehensive analysis of its computational and memory overhead. For computational cost during training, the analysis have been posted in Section 4.5 of the original paper. Furthermore, we conduct an experiment to analyze the computational and memory cost during inference phase on CREMA-D dataset. The baselines include naive MML, PMR, AGM, MLA, ReconBoost. The results are shown in Table 6. We can find that our method achieves superior performance while maintaining competitive inference time. Furthermore, compared with naive MML, our method introduces only 1M additional parameters when 10 classifiers are added. Given the total model size of 23.6M, this corresponds to a relatively small increase of approximately 4%.

Table 6. Computational cost comparison on CREMA-D dataset.

Relevant experiments and discussion will be included in the final version.

Answer for Question "Performance Comparison": From the Table 1 of the original paper, we can find that our method achieves notable performance improvements on both video-audio and tri-modal datasets. Specifically, it improves accuracy by 1.53% on the CREMA-D dataset, 0.10% on the KSounds dataset, 0.27% on VGG dataset, and 0.36% on the NVGesture dataset. However, on the Twitter2015 and Sarcasm datasets, the performance gains are relatively modest, although our method still achieves the best results in most cases. This may be attributed to the limited complementary information between modalities in the text and image data. In fact, we observe that, compared to the best single-modal performance on these datasets, the improvements brought by all multimodal methods are generally minor. Notably, on the Twitter2015 dataset, most multimodal approaches even underperform the best single-modal baseline. Addressing this question is a promising direction for future research.

Reference:
[1]. Hua, Cong, et al. ReconBoost: Boosting can achieve modality reconcilement. ICML. 2024.
[2]. Hu, Peng, et al. Learning cross-modal retrieval with noisy labels. CVPR. 2021.
[3]. Zhang, Xiaohui, et al. Multimodal representation learning by alternating unimodal adaptation. CVPR. 2024.
[4]. Huo, Fushuo, et al. C2kd: Bridging the modality gap for cross-modal knowledge distillation. CVPR. 2024.