Are you SURE? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimation

Regarding the table presentations:

We want to thank you for your constructive feedback about our table presentation. In Table 1 and Table 2, we work with datasets involving only 2 modalities, hence these two tables already cover every possible combination of input modalities. For table 3, which involves 3 modalities, the extended result of this experiment is provided in section A.3.1 owing to space constraint.

In our current settings, we set the missing ratio of the training dataset as $50\\%$, while there are no fixed constraints for testing the dataset. We are working on additional experiments covering other configs with different ratios during training for better empirical results. We will include those results for the camera-ready version.

Thank you once again for taking the time to review our paper. If you have any additional questions, please feel free to reach out to us. We kindly ask you to consider revisiting your evaluation, and we would greatly appreciate your reconsideration to improve the score.

Dear Reviewer dpnx,

We sincerely thank the Reviewer for their time and thoughtful feedback on our paper. Below, we provide detailed responses to each of your questions and concerns:

Regarding comparison with Prompt-based techniques:

Within the limited discussion period, we attempt to put both these suggested methods [1,2] into comparison with SURE for the sentiment analysis task:

As shown, SURE outperforms the two prompt-based approaches in handling missing modalities, achieving better downstream task performance. This advantage may stem from the limited number of learnable parameters introduced by these techniques, which likely constrain their ability to adapt effectively to scenarios with missing modalities.

Additional comparisons with two remaining tasks are currently conducted by us, which will be included if our work is further considered.

[1] Lee, Yi-Lun, et al. "Multimodal prompting with missing modalities for visual recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2] Guo, Zirun, et al. “Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition”. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024.

Regarding the differences between our work and MPLMM:

Our work mainly diverges from this interesting study in several aspects:

• Scope difference: The scope of [2] is focused on Sentimental Analysis and Emotion Recognition tasks, while SURE does not tie in with any assumptions about tasks and types of modalities in the datasets.
• Contribution difference: [2]’s major contribution is the effective prompting technique to generate missing modality features. While they offer robust reconstructed features, they don’t explicitly quantify the reliability of these reconstructions. Their main contribution is not contrast, but supplement our contribution, which offers a method to assess this reliability level beside ordinary reconstruction, and ultimately assess the final output reliability given these reconstructed inputs.

Regarding the evaluation metric:

We intentionally choose PCC over other methods for following reasons:

• Linearity: The enforcement of linear dependence resulting from PCC is indeed desirable. As these relationships are expected to reflect a direct, proportional change, Pearson’s focus on linearity is ideal for assessing whether uncertainty predictions align with the model errors.
• Simplicity and popularity: With the simple linear constraint, we theoretically show the direct relation of PCC loss with log-likelihood loss when the popular Gaussian distribution assumption is being made. This analysis resonates the working and effectiveness of PCC loss in estimating uncertainty.

Regarding the training process:

We separate these two training logics to obtain better stability. While we can jointly train both the reconstruction module and the classifier, the reconstructions during the initial training phase would be unstable and show great variances. The use of such noisy and unstable reconstructions might even hurt or slow down the classifier learning process, and make it more challenging to assess the uncertainty of final outputs.

Dear Reviewer dpnx,

Thank you for your time and effort in reviewing our paper.

We have carefully addressed your concerns and questions throughout this discussion period. As we approach the end of this phase, we kindly ask if you could confirm your evaluation or share any additional feedback you might have.

We greatly value your input and appreciate your thoughtful review.

Best regards,

The Authors

Dear Reviewer DTc9,

We appreciate the Reviewer’s time and effort in evaluating our paper and providing insightful feedback. Below, we address your questions and concerns point by point:

Regarding the motivation of the paper:

The main rationale of our work can be summarized in a single sentence: SURE leverages the strength of pretrained multimodal frameworks on small, incomplete datasets (with missing modalities) for similar tasks, while quantifying prediction uncertainty on these non-ideal samples.

Considering your constructive feedback, we have made several modifications to our Introduction section. These changes better express our motivation in both leveraging pretrained models, as well as assessing models’ reliability when working in such non-ideal scenarios.

Regarding our Figure 1:

We want to clarify that our old Figure 1 in fact highlights the observation that pretrained models converge better and hence achieve better results compared to vanilla models, under the same small-scaled datasets. In this case, both the pretrained and vanilla models ignore the samples with missing modalities in the training dataset.

Regarding the novelty:

We highlight here our unique novelty with this work:

• A plug-in solution to tackle small-scale and incomplete multimodal datasets, this module is training-efficient but yields competitive results and well estimated uncertainty of those results.
• A distribution-free uncertainty estimation technique, which requires no bias about the underlying distribution of prediction
• A unique and intuitive approach to apply Error Propagation in deep learning architectures, effectively separating 2 sources of uncertainty contributing to prediction error: from the model itself and from the reconstructed input.

Modality reconstruction - as suggested, are well-established, we only utilize this strategy to meet our goal of utilizing pretrained frameworks to their best extent.

Regarding the distribution assumption:

We want to clarify that the intention of Equation 4 is a demonstration showing the relevance of minimizing PCC loss between uncertainty and error and minimizing the MSE between standardized versions of these two variables (zero mean and unit variance), but not tie our method with the Gaussian assumption.

Essentially, this relevance shows that our method in fact enforces linear dependence between the two variables after being standardized, hence does not depend on the scale or distribution of these variables. We highlight this point in the Sect. 2.3.

Regarding the use of MMML:

For the introductory presentation, we want to demonstrate the benefit of using pretrained models for smaller datasets of different scales. MMML (Multimodal multi-loss fusion network) [1], is a state-of-the-art fusion architecture for Semantic Analysis on the CMU-MOSI dataset.

[1] Wu, Zehui, et al. "Multimodal multi-loss fusion network for sentiment analysis." Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Regarding to our main source of improvement:

The fact that pretraining contributes to the majority of the performance gain of SURE is not opposed but in companion with our original motivation and methodology.

SURE is designed so that pretrained models can be used to show their effectiveness with small and corrupted datasets, which normally are big challenges to vanilla fusion architectures (as shown in our experiments as you suggested).

SURE’s reconstruction modules are kept simple and not the main focus of our work, and should be thought of conceptually instead, which can be replaced and strengthened. From Table 4, with just a simple reconstruction modules' design (mentioned in Appendix A.1.2), SURE with vanilla fusion framework still shows improved performance, as it still enables effective utilization of every sample (even corrupted) in the training dataset.

Dear Reviewer DTc9,

Thank you for the response. We further highlight our points as follow.

We’d first like to emphasize that, as the Reviewer noted, our work focuses on making the pretrained multimodal models significantly “practical” for downstream tasks that often contain missing modalities. However, naively estimating uncertainty, e.g., using the previous approaches, without co-optimizing the uncertainty objective (the Pearson Correlation objective) stemming from both model nature and missing modalities with the downstream task objective, would not yield satisfactory performance, and therefore we cannot take full advantages of pretrained multimodal models in practice. For example, post-hoc techniques like Monte Carlo Dropout or DeepEnsemble do not work well across 3 different tasks under our extensive investigation (Section 3.3), compared to ours with co-optimization of these two objectives.

More specifically, to tackle this important challenge, our work has provided solid analysis of this phenomenal (via our extensive experiments across tasks) and devised a novel objective function (with well-supported theoretical analysis), as also mentioned by the other reviewers (Reviewers Z2fY, 5FYA). These are the significant contributions of our work toward maximally ensuring the practicality of the pretrained multimodal models in real-world applications, such as in the healthcare domain or self-driving vehicles that usually have missing modalities due to external non-ideal factors during deployment.

Note that, latent modality reconstruction is simply a means for us to best leverage pretrained models and estimate the uncertainty, but is not our main contribution.

For these reasons, we believe that our work is unique compared to all existing approaches, as also pointed out by Reviewers Z2fY, 5FYA.

Dear Reviewer 5FYA,

We thank the Reviewer for spending effort to review our paper and provide valuable feedback. We address your questions and concerns point-by-point below:

Regarding the uncertainty estimation:

We apologize if the previous writing leads to a misunderstanding. What we intentionally want to convey is that our reconstruction logic is the main factor in boosting performance - as it better uses training data and bridges the gap of missing modalities, while uncertainty estimation helps assess the reliability of the result. We correct the Abstract and the Introduction to further clarify this point: Figure 2 is added, which illustrates the interrelation between reconstruction uncertainty, output uncertainty and the final downstream task performance.

Additionally, we illustrate the use of estimated uncertainty to produce better decision-making when deployment by selecting a suitable uncertainty threshold, we can effectively filter out erroneous predictions - Sec 4.3.

Regarding the assumption about the distribution:

We want to clarify these points as follows:

• Our method is a distribution-free technique, essentially meaning we don’t introduce any bias regarding the underlying distribution of estimated output. Our constraint enforces uncertainty to ideally reflect the prediction error via linear dependence regardless of the underlying distribution.
• The Gaussian distribution analysis is the way we connect PCC loss with a common assumption where our loss leads to a solution that is equivalent to the Gaussian solution with more relaxed constraints. However, the closed-form solution of Gaussian assumption cases may not give us the right answer when the underlying distribution is not Gaussian (eg: heavy-tailed distribution).

Regarding the baseline model (IMDer)’s results:

These differences stem from the following differences in the training/evaluation setting:

• Model setup: The original architecture shipped with IMDer is replaced with pretrained MMML model (for the CMU-MOSI task) to ensure similar setup across all baseline methods.
• Training data: 50% sample points of each modality in the training dataset are randomly masked out and considered as missing, compared to the use of full modality samples as in IMDer. The main decrease in performance might be attributed to this reason.
• Evaluation data: Our evaluation involves simulations for different missing modality combinations. We mask out specific modalities over the entire test set and assess the model performance given that corrupted data.

Regarding the evaluation metric:

We intentionally chose the correlation as the evaluation metric due to its ability to capture the linear dependency of these two variables which directly reflects the desirable characteristic of uncertainty. This choice is in alignment with previous series of work in uncertainty estimation literature (e.g. [1]).

For a more general assessment of the uncertainty values produced by SURE, we additionally adopt metrics UCE (Uncertainty Calibration Error) from papers [2] for the three tasks being investigated.

Summary of uncertainty assessment with UCE:

This result suggests that SURE produces good estimation of uncertainty, consistently outperforming representative baselines over the investigated tasks.

[1] Upadhyay, Uddeshya, et al. "Bayescap: Bayesian identity cap for calibrated uncertainty in frozen neural networks." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[2] Guo, Chuan, et al. "On calibration of modern neural networks." International conference on machine learning. PMLR, 2017.

Dear Reviewer 5FYA,

We deeply appreciate the time and effort you have devoted to reviewing our paper.

During this discussion period, we have carefully addressed your concerns and questions to the best of our ability. As the discussion phase nears its conclusion, we kindly ask if you could confirm your evaluation or provide any additional feedback.

Thank you again for your thoughtful review and valuable input.

Best regards,

The Authors

Dear Reviewer Z2fY,

We appreciate you taking the time to review our paper and provide valuable feedback. Please find detailed responses to your questions and concerns below.

Regarding the choice of baselines:

We deliberately select methods that address missing modalities through reconstruction as our baselines. This approach is suitable to incorporate into pretrained multimodal pipelines, as it requires no architecture/model change, hence effectively leveraging the strengths of the pretrained models - which align with our primary motivation. Additionally, using these baselines enables a more meaningful comparison, as the same pretrained pipelines can be applied uniformly across all methods. The chosen baselines are also current SOTAs in reconstruction-based techniques, covering both raw data reconstruction (IMDer, DiCMoR - 2024) and latent reconstruction (ActionMAE - 2023), which are two main directions of this approach.

Within this limited discussion period, we additionally compare SURE with two recent works applying Prompt-based techniques [1,2], which are also designed to leverage pretrained models and tackle missing modalities - Thanks to Reviewer dpnx’s suggestions. In summary, the results for Semantic Analysis task suggests slightly better performance of SURE over these methods, highlighting our efficiency in dealing with missing modalities. We kindly refer the Reviewer to our response to Reviewer dpnx, or to our revised Appendix A.3.3 for additional details.

[1] Lee, Yi-Lun, et al. "Multimodal prompting with missing modalities for visual recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2] Guo, Zirun, et al. “Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition”. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024.

Regarding our writings:

We have made following major changes in this rebuttal phase to better explain our ideas and concepts:

• Introduction: We additionally modify to better introduce the context and problem: Tacking small datasets with missing modality by leveraging pretrained networks, assess their reliability via uncertainty estimation.
• Method: we provide additional reference to Appendix, where details about different modules and architecture choices are well defined and analyzed (including reconstruction modules).
• Evaluation: we provide reference to Appendix, where details about the integration of SURE with pretrained frameworks are discussed. Text modifications regarding additional experiments are also made.

We hope these changes satisfactorily explain our method. We are happy to further discuss and clarify if needed.

Regarding the choice of modal absence scenarios:

There are several reasons for us to tackle intermodal missing:

• We follow the majority of literature dealing with multimodal missing, which all focus in intermodal absence settings.
• Intramodal absence is in fact missing data in unimodal settings - there are already many different lines of work extensively investigating this problem: data imputation, reconstruction, …

That being said, while sticking to intermodal absence, we already target the most generic experiment setting that covers most real-world scenarios:

• Missing modalities can presented in both training or evaluation phases.
• Different input combinations, essentially cover every combination of missing modalities (When only a single modality is presented - Main text experiments, every possible combinations - Appendix A.3.1.)

We’re open to consider running any additional experiments the Reviewer suggests to better improve our paper.

Regarding the choice of MMML:

The use of MMML is just for demonstration purposes. MMML [3] is a state-of-the-art fusion architecture for Semantic Analysis on the CMU-MOSI dataset. The main goal of this toy experiment is to show the benefit of using pretrained models for smaller datasets of different scales.

[3] Wu, Zehui, et al. "Multimodal multi-loss fusion network for sentiment analysis." Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Once again, thank you for reviewing the paper. We think we have solved your problem as much as possible in rebuttal, If you have any further questions, please do not hesitate to contact us. If possible, we would like to thank you for reconsidering to improve your score.