Deep Correlated Prompting for Visual Recognition with Missing Modalities

Reviewer# YNgk

1. Novelty compared to MMP

Many thanks for your question. MMP has first introduced prompt learning to handle the missing-modality setting. It inserts learnable tensors, i.e., prompts at each layer which still keeping the image encoder and text encoder fixed to guide the model to fit missing-modality cases. However, the inserted prompts of MMP across different layers are independent, while we believe that the prompts across different layers and various modalities can provide beneficial information for each other to better fit missing-modality cases. Thus, we propose correlated prompts which generate the prompts of the next layer based on the prompts of both modalities in the current layer. For dynamic prompts, our intuition is that the prompts proposed by MMP are fixed for different inputs during inference and fail to fit the missing cases of different inputs, and we thus propose to dynamically compute the prompts based on different input features to better guide the behavior of the model. This procedure is implemented by a self-attention layer with a randomly initialized tensor as a query and the input features as keys and values. Besides, we propose modal-common prompts which store the shared information across different modalities, which can complement the model with common information across different modalities and facilitate the model to encode modal-specific information to better handle the missing scenarios in each modality.

2. Performance v.s. MMP when owning comparable parameters

We compare our method with MMP by allowing them to own comparable parameters. Specially, either when we decrease the required parameters of our method to 0.2M (0.2% of the entire model) by reducing the channel dimension, or we simultaneously increase the required parameters of both methods, our method consistently outperforms MMP, which verifies its effectiveness.

1. Ablation studies by using other multimodal backbones

We provide the results by comparing our method with the baseline method upon the single-stream ViLT backbone, and also comparing them upon the two-stream CoCa backbone as below.

We first provide the results on the ViLT backbone. Our method large outperforms the baseline across different settings on three datasets.

We offer the results on the CoCa backbone. We can find that our method achieves superior performance than the baseline.

2. Performance when either image or text modalities are completely missing in Table 4

We provide the results as follows.

3. How modal-common features are disentangled

Specifically, we introduce a shared learnable prompt across different modality encoders which embeds modal-common information for different modalities. To transform it into the feature space of various modalities, we introduce an independent projection function for each modality to change the channels of the shared learnable prompt. It explicitly embeds modal-common information and inversely encourages other prompts to provide modal-specific information to guide the model to handle different missing-modality cases.

4. Show the layer J-th in Figure 1 (framework overview)

Thanks for your comments. We will update the J-th layer in our manuscript.

5. Phrase correction.

Many thanks for your advice, we will update the expressions.

1. Efficacy of each proposed prompt.

We place each proposed prompt upon the baseline method, and show the results as below on the MMIMDb dataset upon the missing-both setting with η=70%. It’s observed that each proposed prompt could notably boost the performance.

2. Visualization of each learnable prompt.

We offer visualizations for the dynamic prompts using the T-SNE method on the Food101 dataset upon the missing-both setting with η=70% and η=50%. As Food101 have 101 classes which is redundant for visualization, we select a subset with 11 classes to give better visualization results. It’s observed that the dynamic prompts could roughly categorize the prompts into distinct classes. Besides, for different missing settings with η=70% and η=50%, the distribution of dynamic prompts with respect to the same inputs are different, which show that dynamic prompts can learn to generate various expressions for each input upon different missing settings.

3. Exponential increased prompts when more modalities are introduced.

For M input modalities, it requires $2^M-1$ types of prompts for all missing-modality cases. This may be redundant, but we expect a specific prompt to handle each missing-modality case to better guide the model to handle different missing scenarios. Besides, the overall extra consumed parameters are quite few as the modalities increase as the projection functions are shared across different modalities. Thus as the modalities increase, we only introduce new learnable prompts which occupy quite few parameters. For example, for each missing-modality case, the newly introduced prompts own 768*36=27648≈0.03M parameters with a prompt length of 36 for each modality encoder.

4. Identical values in Table 4 compared with MMP

Sorry for the errors. We wrongly set the values when organizing the tables from different resources. The correct values should be 48.23 for missing rate = 70% (100% image and 30% text) in MM-IMDb, 61.12 for missing rate = 70% (100% image and 30% text) in Hateful Memes, and 63.24 for missing rate = 70% (30% image and 100% text) in Hateful Memes. We will current them in the manuscript.

5. Results on single-stream backbones

We compare our method with MMP based on the ViLT backbone. We don’t use features of two modalities to generate prompts of the next layer as Eq.5. The results on three datasets with η=70% upon different missing settings are shown below. It’s observed that our method shows superior performance than MMP.

1. Proposed prompts not directly connected to the missing modality problem.

Sorry for the mis-clarification in the manuscript to mislead you. In line 146-150 of our manuscript, we state that we set different prompts for various missing modalities. Specifically, for correlated prompts, we independently set the initial prompt of the first layer for different missing-modality settings, which will be transformed via a projection function to generate the prompts for the next layer. For the dynamic prompts, we initialize different queries for different missing modalities, which are then used to compute the dynamic prompts based on input features in a self-attention manner. For the modal-common prompts, we initialize different modal-common prompts for different missing-modality settings, which will be transformed into embedding space via a projection function to provide prompts. Sorry for missing the expressions for the dynamic prompts and modal-common prompts upon different missing-modality settings. We will update them in the manuscript to give better clarifications.

2. Effectiveness regarding the correlated prompts compared to MMP.

Based on the same CLIP backbone, we compare the performance of MMP to our proposed correlated prompts on the MMIMDb dataset with different missing ratios under the missing-both case as below. It’s observed that our proposed correlated prompts consistently offer superior performance than MMP under the same setting.

3. Comparison between the integration of MMP and ours.

We integrate the proposed dynamic prompts and modal-common prompts into MMP based on the CLIP backbone to form a method termed as MMP*, and compare our method with it on the MMIMDb dataset with different missing ratios under the missing-both case as below. It’s observed that our method notably outperforms MMP* with different η.

4. Analysis between parameters v.s. performance

We first give an ablation for the parameters of each proposed prompt. We give the performance with η=70% on the MMIMDb dataset upon the missing-both setting. As shown below, as we insert each proposed prompt, the performance consistently increases. The proposed prompts overall bring extra 4.0M trainable parameters, which is only 2.4% of the overall framework.

Besides, we test the relationships between the brought extra parameters and performance, and show the results as below. It’s observed that the performance continues to increase when the prompt depth ranges from 12 to 36, and reaches a peak when the prompt depth equals 36. The parameters consistently increase as the prompt length grows.

Finally, we compare our method with MMP with the same parameters. It’s observed that with the same parameters, our method consistently achieves better performance.

5. The dynamic prompting technique is not new.

Many thanks for your question. The dynamic prompts have been previous investigated in other methods. However, dynamic prompts in the missing-modality scenarios have not been studied. It’s worth exploring whether generating various prompts according to different missing cases and input features can well guide the model to fit the missing-modality settings. Our manuscript verifies the efficacy of this design.

1. The mechanism of different prompts.

Many thanks for your question. We have plotted a figure to further illustrate our proposed prompts by comparing them with our baseline and MMP[17], which can be found in the pdf file of Author Rebuttal. The baseline simply uses fixed image encoder and text encoder and only finetunes the classifier to handle downstream tasks. To well adapt to missing modalities, MMP[17] inserts learnable tensors, i.e., prompts at each layer which still keeping the image encoder and text encoder fixed to guide the model to fit missing-modality cases. However, the inserted prompts of MMP across different layers are independent, while we believe that the prompts across different layers and various modalities can provide beneficial information for each other to better fit missing-modality cases. Thus, we propose correlated prompts which generate the prompts of the next layer based on the prompts of both modalities in the current layer. For dynamic prompts, our intuition is that the prompts proposed by MMP are fixed for different inputs during inference and fail to fit the missing cases of different inputs, and we thus propose to dynamically compute the prompts based on different input features to better guide the behavior of the model. This procedure is implemented by a self-attention layer with a randomly initialized tensor as a query and the input features as keys and values. Besides, we propose modal-common prompts which store the shared information across different modalities, which can complement the model with common information across different modalities and facilitate the model to encode modal-specific information to better handle the missing scenarios in each modality.

2. What makes a "dynamic" prompt "dynamic"?

We propose to dynamically compute the prompts based on different input features, which avoids employing fixed prompts for different input features during training and inference. This procedure is implemented by a self-attention layer with a randomly initialized tensor as a query and the input features as keys and values. You can view the illustration in the figure of Author Rebuttal.pdf.

3. The baseline setting.

The baseline setting is using the text encoder and image encoder to encode the input texts and images, whose output features are fed into the classifier for recognition. In this procedure, only the classifier is updated and other model components including the text encoder and the image encoder are kept fixed. The only difference between our method and the baseline is inserting learnable prompts at each layer which only bring few extra parameters. Specifically, when a modality is missing, we simply don’t feed the corresponding features into the modality encoder for processing and set the outputs of this modality encoder as zeros. Overall, we have tried three baseline settings, including (1) inputting zeros for the modality encoder when a modality is missing, (2) inputting the average of the modality tokens for the modality encoder when a modality is missing, and (3) not feeding input features for the modality encoder when a modality is missing and setting the outputs of this modality encoder as zeros (Default). As shown below, out experiments on the MMIMDb dataset with η=70% show the last choice gives best performance, and thus we adopt it as a stronger baseline.

4. The re-implementation setting of MMP.

MMP adds learnable prompts with a length of 16 at the bottom 6 layers based on the ViLT backbone. we re-implement it based on the CLIP backbone by adding learnable prompts with a length of 16 at the bottom 6 layers in the image encoder and text encoder, respectively. Our method owns the same number of inserted layers and prompt length with MMP. The only difference is that we insert three proposed prompts into the model.

5. Comparison with MMP using the ViLT backbone

We compare our method with MMP using the ViLT backbone and show the results on three datasets with η=70% upon the missing-both setting as below. It’s observed that our method shows superior performance than MMP. It’s also noticed that using ViLT as the backbone achieves inferior performance than CLIP, and thus we adopt CLIP as the default backbone.

6. Zero-shot performance of CLIP.

We test the performance of zero-shot CLIP on the MMIMDb, Food101 and Hateful Memes datasets. We first calculate and store the averaged embeddings of all classes of both modalities on each dataset. When the input modalities are complete, we calculate the similarities between the output features and the pre-computed class embeddings for each modality encoder. We select the class with the highest similarity as the recognition output within both modalities. When a modality is missing, we select the class whose embedding owns the highest similarity with the output features of the available input modality as the recognition output. We compare the performance of zero-shot CLIP, finetuned CLIP (our baseline, which sets output features as zeros when a modality is missing) and our method as below. The zero-shot CLIP achieves inferior performance than the other methods, and ours perform best. We suppose that the zero-shot CLIP not finetuned on the downstream datasets fails to well adapt to downstream setting. Finetuning the CLIP could notably increase the performance, and ours further boost the performance by injecting different kinds of learnable prompts.