Facilitating Multimodal Classification via Dynamically Learning Modality Gap

Thanks for your comments.
Overall Response: Our motivation is to explore the relationship between label fitting (one-hot label) and modality imbalance, thus to find a way to mitigate the impact of label fitting. We first utilize a toy noise label experiment to demonstrate that label fitting can lead to modality imbalance. Specifically, we compared the classification performance by using the original one-hot label loss ($L_s$), a label free loss ($L_u$) and their linear combination loss. One can image that $L_s$ is a supervised loss while $L_u$ is a unsupervised loss. The results in Figure 1 of the original paper show that even the overall classification performance of $L_u$ is worse than that of $L_s$, $L_u$ has the smallest modality performance gap, indicating that the unsupervised loss could mitigate the modality imbalance but lead to low classification performance. This inspires that there may be an optimal trade-off between the supervised loss $L_s$ and unsupervised loss $L_u$. To this end, one simple way is to tune the $\alpha\in [0,1]$ for $\alpha L_s + (1-\alpha)L_u$. We finally found that $0.7L_s+0.3L_u$ has the best overall classification performance with the middle modality performance gap, compared with $L_s$ and $L_u$. Based on the experimental findings, we then employ the contrastive unsupervised learning to align the modality representation of multimodal data [1], thereby mitigating the modality imbalance. In this way, overall classification performance of MML can be improved. It is worth to mention that the proposed method is a generic algorithm that can be integrated with many unsupervised learning.
1. Answer for Question "Insufficient Analysis of Modality Imbalance": The overall classification performance of MML is a primary concern to us and is affected by the modality imbalance phenomenon. Literature [2,3,4] theoretically reveals the intrinsic relationship between modality performance gap and modality imbalance, i.e., the larger the modality performance gap, the more serious the modality imbalance. Based on these theoretical conclusions and our findings, we deeply explore the impact of label fitting on modality imbalance in this paper. The main logic of our paper is that label fitting leads to the modality performance gap, which in turn exacerbates the modality imbalance phenomenon. As you mentioned, increasing the weight of noise labels to reduce the impact of label fitting is positively correlated with a reduction in the modality performance gap. Meanwhile, as the modality performance gap narrows, the modality competition phenomenon will be alleviated [3], thus alleviating the modality imbalance. In summary, the overall performance of MML is improved through intervening the label fitting.
2. Answer for Question "Difference Between Mixed Loss and Label Smoothing": The mixed loss is the label smoothing. As mentioned in the overall response part, the experiment of label noise aims to explore the relationship between label fitting and modality performance gap, thereby affecting overall performance. Therefore, we did not discuss the mixed loss and label smoothing in the paper. We will add appropriate discussion to the final version.
3. Answer for Question "Practicality of Dynamic Weight Methods": Heuristic method is designed primarily to validate the motivation behind our methods. For example, by carefully adjusting the function form, such as using the polynomial function $f(t) = a t^3 + b t^2 + c t + d$, with coefficients $a = 1.5 \times 10^{-4}$, $b = -6.5 \times 10^{-3}$, $c = 3.2 \times 10^{-2}$, $d = 1$, the classification accuracy achieves 71.22% compared to 70.04% with the MLA on the Kinetics-Sounds dataset.
In fact, the complexity of bi-level optimization [5] and our algorithm is O(n). More details of bi-level optimization can be referred to literature [5]. To further demonstrate the efficiency of our method, we adopt MLA and our method for experiments and report the training time cost on Kinetics-Sounds dataset. The average time costs for MLA and our method are 327s and 301s for an epoch, respectively. Therefore, we can demonstrate the practicality of our method. We will add some discussion and results in the final version.
References:
[1]. Fan, Yunfeng, et al. Detached and Interactive Multimodal Learning. ACMMM, 2024.
[2]. Zhang, Xiaohui, et al. Multimodal Representation Learning by Alternating Unimodal Adaptation. CVPR, 2024.
[3]. Huang, Yu, et al. Modality competition: What makes joint training of multi-modal network fail in deep learning? (provably). ICML. 2022.
[4]. Wang, Wei, and Zhi-Hua Zhou. Co-training with insufficient views. ACML. 2013.
[5]. Vicente, Luis N., and Paul H. Calamai. Bilevel and multilevel programming: A bibliography review. JGO, 1994.

Thank you for your comments.
1.Answer for Question "Closely With the Learning Strategy Curve": We use the polynomial function $f(t) = a t^3 + b t^2 + c t + d$, with coefficients $a = 1.5 \times 10^{-4}$, $b = -6.5 \times 10^{-3}$, $c = 3.2 \times 10^{-2}$, $d = 1$ to fit the current learning curve and represent $\omega(t)$. The results in Table 1 indicate some degree of improvement. However, the functions fitted to different datasets are different, and it is difficult to find a unified function form, so we put forward the learning-based method.

Table 1. Performance of different weights on Kinetics-Sounds.

Thank you for your comments.
1. Answer for Question "Contrastive Learning can Mitigate Modality Imbalance": Our motivation aims to find a way to mitigate the modality performance gap caused by label fitting. As shown in literature [1], different modalities will compete with each other during the joint-training. Specifically, the neural network will not efficiently learn all features from different modalities, and only a subset of modality encoders will capture sufficient feature representations. Meanwhile, contrastive learning aims to learn similar representations for data pairs of different modalities [2], thus aligning the multimodal representations for all modalities. Hence, we utilize the contrastive learning to intervene the label fitting. Furthermore, this intervention is dynamic as the learning process. In the initial stage, the model primarily focuses on feature learning. As training progresses, label fitting becomes increasingly important. This phenomenon aligns with our expectations.
2. Answer for Question "Lack Comparison with SOTA Method": We conduct a comparison with existing state-of-the-art methods. The results in Table 1 demonstrate that our method achieves higher classification accuracy compared to these methods. Notably, our approach significantly narrows the modality performance gap, which contributes to an improvement in overall classification performance. We will add more detailed analysis in the final version.

Table 1. Performance comparison between state-of-the-art methods and our method on Kinetics-Sounds

3. Answer for Question "Results using CLIP Pre-trained Encoder": In this case, following the setting of [7], we use the pre-trained weights in the CLIP as initialization. In the subsequent training process, the feature extractor and classification head are fine-tuned. We will add more details in the final version.
References:
[1]. Huang, Yu, et al. Modality competition: What makes joint training of multi-modal network fail in deep learning?(provably). ICML. 2022.
[2]. Fan, Yunfeng, et al. Detached and Interactive Multimodal Learning. ACMMM, 2024.
[3]. Du, Chenzhuang, et al. On Uni-Modal Feature Learning in Supervised Multi-Modal Learning. ICML 2023.
[4]. Hua, Cong, et al. ReconBoost: Boosting Can Achieve Modality Reconcilement. ICML 2024.
[5]. Zhang, Qingyang, et al. Provable Dynamic Fusion for Low-Quality Multimodal Data. ICML 2023.
[6]. Wei, Yake, et al. MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance. ICML 2024.
[7]. Zhang, Xiaohui, et al. Multimodal Representation Learning by Alternating Unimodal Adaptation. CVPR, 2024.

Thank you for your detailed comments.
1.Answer for Question "Performance on Larger Dataset": We compare our method with the state-of-the-art method MLA on the VGGSound dataset using the experimental settings described in the literature [1]. Experimental results in Table 1 demonstrate that our method achieves higher accuracy on larger datasets compared to state-of-the-art method MLA.

Table 1. Performance comparison between state-of-the-art method and our method on VGGSound dataset.

2. Answer for Question "Restriction of Strong Modality": Table 4 in the original paper shows that, compared to the standard joint training method, both the strong and weak modalities have shown improvement, with the weak modality improving more than the strong modality. Additionally, the single-modality performance of combined training is close to that of individual training.
References:
[1]. On Uni-Modal Feature Learning in Supervised Multi-Modal Learning. ICML 2023.