Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation

Thank you very much for your review. We thank you for acknowledging the strengths, simplicity and performance of our method. We respond to your other comments below. 

1. The method proposed in this paper may not be directly applicable to scenarios where the training set has missing modalities.
   This is a good point. Indeed if a modality is completely missing from the training set, then our method (and almost all existing methods) cannot compensate for that. However, if the training set has some samples missing for some modalities, we can pretrain a network with the available modalities and then perform parameter-efficient network adaptation to compensate for the missing modalities at the test time. Since our approach is mainly focused on adapting a pretrained model to different missing modality situations, we can use the same adaptation procedure using a different base network. The performance in this case will probably depend on the quality of the base model pretrained with missing samples from different modalities.

2. The method proposed in this paper lacks extensive comparisons with other methods~(for example, [1,2,3]) in the context of multimodal classification tasks, which significantly undermines the persuasiveness of Table 5.
   Our work focuses on parameter efficient adaptation techniques for existing multimodal models. The main goal is to adapt already pretrained models to different missing modality scenarios. While we cited the first two papers ([1, 2]) in our work, we did not compare with them for the following reasons:
   
   Paper [1] suggested multitask learning and search for optimal fusion strategy to enhance robustness (mainly for two modalities). Extending the methods for masking out attention for more than two modalities and for tasks that rely on all the tokens is not obvious. Furthermore, the paper did not provide a codebase or results for the datasets we used in our experiments; therefore, we could not compare with them.
   
   Paper [2] suggested Shared Specific Feature Modelling (ShaSpec), which is based on modeling and fusing shared and specific features. The paper focused on medical imaging (brain tumor segmentation) and classification. They did not provide any baseline for the datasets and tasks that we used in our work. That is why we did not compare our results with the paper.
   
   Paper [3] - This paper was submitted to arxiv 12 days after the ICLR submission deadline. For this reason, we could not cite or compare with this paper. We will be happy to cite the paper in the revised version. The paper did not provide code or baseline results for the datasets and tasks that we used in our paper. For these reasons, we are unable to provide a comparison.

6. There is no study that fairly compares the parameter efficiency of the proposed method with competing methods?
   Table 4 provides comparison of different parameter-efficient adaptation methods that can be used in our proposed framework. We compared the performance of our adaptation-based method with existing robust methods in Table 2 and 3.  Since these methods are primarily designed to be robust to missing modalities, their emphasis is not necessarily to learn parameter-efficient networks. For this reason, we felt a comparison in terms of parameter efficiency would be unnecessary or unfair. We are listing the total number of parameters for some of these models here for completeness. 

   Method	Total Number of Parameters (M)
   CEN	118.2
   MDRNet	64.60
   RTFNet (ResNet-152)	254.51
   SAGate	110.85
   AsymFussion (ResNet-101)	118.2
   Ours (Adapted CMNext Model)	117.3 (Base Model: 116.56M, Adaptation Layers: 0.789M)

7. What are the possible reasons for relatively lower performance of other adaptation strategies, compared to SSA, in Table 4? Some explanation would be helpful to better understand their comparison in the missing modalities scenario.
   SSF learns parameters ($\gamma$, $\beta$) for scaling and shifting the intermediate features. We can consider BitFit as a subset of SSF which only shifts the intermediate features. LoRA is only applied to the attention layers ($W_q$ and $W_v$) following the original paper and the number of parameters for LoRA is less than SSF. Norm only optimizes the norm layers while SSF is applied after every linear, convolutional and norm layer. Which gives SSF more advantage over other methods to modulate the intermediate features and perform better on downstream tasks.

Thank you very much for your insightful and detailed review. Thank you for nicely summarizing the strengths and contributions of our paper. We respond to your questions and other comments below. 

1. Although the idea is simple and effective for achieving parameter-efficient adaptation of models, the application of existing techniques (i.e. low rank adaptation, scale and shift transformations, and BitFit) to adapt a pre-trained model on a subset of modalities seems low from a novelty aspect.
   We agree with you that the idea is simple and effective. Low rank adaptation, BitFit and Scale and Shift are originally designed for efficient model finetuning and transfer learning. We show that they can also be applied to adapt models to different missing modality scenarios.
   1) To the best of our knowledge, we are the first to show that parameter efficient adaptation can be used to enhance performance in different missing modality scenarios. While it may be simple, we hope this will be informative and useful for the broader community.
   2) We provide a generic framework for missing modality adaptation that can be applied to a wide range of multimodal datasets, tasks, and models.
   3) The adapted model can easily switch to different network states based on the available modalities with minimal latency, computational, or memory overhead.

2. There is no insight and/or analyses into how the proposed parameter-efficient is helpful towards making the model robust to missing modalities. It is important to develop the grounding of the proposed method since only showing improvements over the baseline with empirical results doesn't help much towards this.
   Our main motivation and insight for the proposed approach is to adapt the functional representation of the network according to the available modality combination. For instance, if one of the modalities is missing, we want to modify the feature representations and fusion of other modalities. We could do this by changing the entire network for every modality combination, which is infeasible due to computational and storage cost. In this paper, we perform parameter-efficient adaptation to achieve the same goal. A rigorous analysis of how the network adaptation changes the functional representation of the network will certainly be great, but it is a nontrivial task and beyond the scope of one paper.

3. The paper doesn’t mention any clear justification on why SSA was chosen over the LoRA and BitFit to report the results in the paper.
   We have discussed this in the second paragraph of Section 3.3. We primarily selected the SSF technique because of its simplicity and effectiveness. SSF has several benefits over other adaptation methods:
   1) The parameters (γ, β) are independent of the input features, which makes SSF applicable to diverse tasks and input modality combinations. 
   2) We can easily insert SSF layers in the existing model without changing the model architecture.
   3) We can easily switch/select the corresponding SSF parameters for a given input modality combination.
   4) SSF works relatively better than LoRa, BitFit and Norm in most of the scenarios as shown in Table 4.

Thank you for your detailed and insightful review and acknowledging the strengths of our paper. Below we respond to your other comments and questions. 

1. The method only uses zero to replace missing modalities, lacking other compensation methods such as duplicating available modal data or utilizing cross-modal generation techniques to fully showcase the impact of missing modalities.
   We used zeros to replace missing modalities because that seems to be a common practice in recent papers [1, 2]. We compared the performance of duplicating available modalities and replacing zeros for missing modalities in the table below (same settings as Table 1 in the paper). The results show that duplicating available modalities is inferior to the adapted model (and in some cases worse than replacing zeros for missing modalities). For MFNet and NYUDv2 datasets, we replaced the missing modality with a copy of the available modality. For the MCubeS dataset, we replaced the missing modalities with a copy of RGB modality. In all the combinations, the adapted model works much better than the modality duplication method.

   Dataset	Input	Missing	Pretrained (Zeros for missing)	Pretrained (Duplicate for missing)	Adapted
   MFNet	RGB-Thermal	-	60.10	-	-
   	RGB	Thermal	53.71	52.33	55.22
   	Thermal	RGB	35.48	44.43	50.89
   NYUDv2	RGB-Depth	-	56.30
   	RGB	Depth	51.19	46.19	52.82
   	Depth	RGB	5.26	13.94	36.72
   MCubeS	RGB-A-D-N	-	51.54	-	-
   	RGB-A-D	N	49.06	49.93	51.11
   	RGB-A	D-N	48.81	49.23	50.62
   	RGB	A-D-N	42.32	48.96	50.43

   Utilizing cross-modal generation is also a very good idea for missing modality compensation. Table 3 in our paper provides a comparison with one such method, TokenFusion, which dynamically detects uninformative tokens and substitutes these tokens with projected and aggregated tokens from available modalities. The CRM method in Table 2 uses self-distillation between the clean and masked modalities to learn complementary and non-local representations for better performance on missing modality scenarios. Our method performs better than both these methods in most of the scenarios.

2. The method's SSF strategy is an input-independent adaptation approach; it struggles to maintain reliable performance in scenarios with significant distribution differences between the training and testing sets.
   This is a good point and SSF may perform poorly in some cases, but in our experience SSF provides quite robust adaptation. According to [3], parameters learnt by SSF are input-independent that makes it more suitable for representing the distribution of the different downstream dataset by modulating the intermediate feature. We could learn the modulation parameters by conditioning them on the input features, but that would require some additional modules (e.g. MLP and activation functions) at the expense of additional parameters.

[1] Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, Xi Peng; “Are Multimodal Transformers Robust to Missing Modality?”, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 18177-18186

[2] Sangmin Woo, Sumin Lee, Yeonju Park, Muhammad Adi Nugroho, Changick Kim; “Towards Good Practices for Missing Modality Robust Action Recognition.”, AAAI 2023: 2776-2784

[3] Lian, Dongze and Zhou, Daquan and Feng, Jiashi and Wang, Xinchao; “Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning”, Advances in Neural Information Processing Systems (NeurIPS), 2022

Thank you for your review. We address your comments below. 

1. The math behind this paper is not clear.
   The main problem formulation is presented in Sec. 3. We believe we provided a concise mathematical description of our approach without unnecessary details. Equation 1 can be viewed as a model/network that is trained for all modalities $\mathcal{M}$ (consider it a baseline model). Equation 2 can be viewed as a model that is trained for a subset of modalities $\mathcal{S}$. We can in principle train one such model for every possible modality combination (dedicated training), but that would be prohibitive with a large number of possible modality combinations. Equation 3 represents our approach where we can learn a small set of parameters for different missing modality situations to adapt an already trained model. Mathematical formulations for different adaptation techniques are presented in Sec. 3.2. Please let us know if you have any other questions, and we will be happy to elaborate further.

2. The author does not give a reason why the data with missing modalities is not considered in the training phase.
   It is unclear what you mean by this comment. We suspect you mean that we could use data with missing modalities at the training stage and either train a single robust network that performs well with all possible missing modalities or train one network for every specific missing modality combination. The latter approach is obviously infeasible as the number of networks required to train can become large. The former approach is something we considered and compared against (see results in Table 2 for VPFNet, SpiderMesh, CRM and Table 3 for TokenFusion). Our proposed network adaptation method clearly outperforms a single network trained to perform well with all possible missing modalities.
   In our experiments, we wanted to separate the effects of robust training and network adaptation, but our proposed network adaptation scheme can be easily combined with a network pretrained in a robust manner. Furthermore, we compared our method with CRM in Table 2 that is trained in a robust manner and performs worse than our method on an average.

3. The datasets are not diverse enough. The authors only considered segmentation and sentiment analysis problems, different types of data should be considered.
   We performed experiments for 3 different tasks (semantic segmentation, material segmentation and sentiment analysis). The tasks involve 5 different datasets and a total of 8 different modalities (RGB images, thermal images, depth maps, angle of linear polarization, degree of linear polarization, video, audio and text). For semantic segmentation we considered 2 different datasets: MFNet (RGB-thermal) and NYUDv2 (RGB-depth). For material segmentation we considered the MCubeS dataset that has four modalities: RGB images, thermal images, angle of linear polarization and degree of linear polarization. Multimodal sentiment analysis datasets (CMU-MOSI and CMU-MOSEI) have video, audio and text as modalities. We believe these datasets are diverse enough for a single paper. Please let us know if you have any specific dataset or task in mind. We will be happy to address that.

4. Figure 1 is not clear.
   Figure 1 illustrates our overall approach to missing modality adaptation. Figure 1(a) shows that we can use a pretrained model, freeze all the weights, and insert adaptable layers after linear, convolutional, and norm (both batchnorm and layernorm) layers. Frozen and adaptable layers are illustrated with different colors for better understanding. Figure 1(b,c) shows two types of parameter-efficient adaptation schemes that can be used to modify the network output. We explained the figure in the first paragraph of Section 3.3 and explained the overall approach in Section 3.1 Equation 3. Please let us know which part is still not clear and we will explain or modify it as necessary.

Dear Reviewer,
We hope you had a chance to read our response to your questions. We appreciate your feedback. Please let us know if you have any additional question that we can answer today. Thank you for your kind attention.