Probabilistic Conformal Distillation for Enhancing Missing Modality Robustness

Q1: First, the whole framework relies on a definition of positive and negative points, which is not obvious for a lot of multi-modal learning problems (For example, cross-modal retrieval, missing-modal imputation, etc). Second, as the loss $L\_g$ seems to scale quadratically with batch size, the time and memory complexity of PCD need to be formally analyzed.

• We are sorry that there may be somewhat misunderstanding about the mission of PCD. PCD focuses on robust multimodal fusion and requires that the maximum number of input modalities be greater than 1. Therefore, mainstream cross-modal retrieval and generation tasks, which typically have only a single input modality, are not suitable for the missing-modality problem explored by PCD. As for the recent popularity of the multimodal retrieval task, where the query text prompts modifications in the query image, will become invalid if any input modality is missing. Hence, it does not align with the problem that PCD aims to solve. Besides, although PCD relies on the definition of positive and negative points, class labels are not necessary. When class labels are available (e.g. the classification tasks), modality-complete representations sharing the same class with the modality-missing input are considered positive points, while the ones from different classes are negative points. When class labels are unavailable (e.g. the segmentation tasks), only the modality-complete representation of the same sample is considered positive, and all others are considered negative, as we have set up in the segmentation task.
• To address the concern about computational cost, we estimated the memory complexity and the time for each iteration of PCD and the other three methods with the same batchsize (64) on CeFA. It can be seen that PCD does not significantly increase training time and memory complexity.

In addition, we explore the relationship between batch size and memory complexity, and computation time. The results indicate that a larger batchsize is not always better. The optimal batchsize (64) has a relatively low computational cost.

Q2: For segmentation, the improvements seem marginal at best, therefore, it is important to provide error bars to validate whether there is any statistically significant difference.

In Table 6 in Appendix, we detail the stability experiments for PCD across four datasets. Each experiment is repeated for three times, allowing to calculate the average score along with the standard deviation. The results reveal that, even in its worst-case scenario, PCD outperforms the best competing methods. These outcomes not only underscore PCD’s superior performance but also attest to its stability and consistency across a wide range of segmentation testing conditions. In addition, to further confirm the effectiveness of PCD on segmentation tasks, we conduct experiments on a larger dataset, SUN RGB-D[1], which contains 5,285 RGB-Depth pairs for training and 5050 pairs for testing. The results are shown below. We can see that PCD is effective even on a larger segmented dataset.

Q3: For segmentation, Lu, just encourages the latent representation of all samples to be distinctive from each other, and I do not see the purpose of Lg, as there is nothing special about the inter-class geometry. It would be important to conduct ablation studies for segmentation, to see whether the inclusion of these losses make any difference.

We are sorry for the misunderstanding caused by the lack of clarity. PCD aims to model different modality-missing representations as distinct distributions to fit their unknown PDFs. This is realized by considering properties of positive and negative points on the modeled distribution $q(z|\mathrm{x}\_i)$. It is important to note that all positive and negative points are generated by a pretrained modality-complete model and remain fixed during training. In a segmentation task, for a modality-missing input, there is only one positive point $z_p^\*$ for the PDF of the input's mapping variables in the modality-complete space, namely, its corresponding modality-complete representation. Here, the negative points $z_n^\*$ are modality-complete representations of all other samples. The loss $L_u$ is used to maximize the probabilities of $q(z|\mathrm{x}_i)$ at $z_p^\*$ and minimize them at $z_n^\*$. This implicitly encourages the mean of $q(z|\mathrm{x}\_i)$ closer to $z\_p^\*$ and further away from $z\_n^\*$. Besides, in $L\_g$, the structures are represented by calculating the distances of peak points of $q(z|\mathrm{x}\_i)$ and $z^⋆$, respectively, where $z^⋆$ is no longer divided into $z\_p^\*$ and $z\_n^\*$. The complete ablation studies is shown in Table 7 in Apendeix or Table 1 in the pdf (the same), which verify the effectiveness of $L\_u,L\_g$.

Q4: There are some writing issues that need to be addressed. Equation (1) is about the modal of a distribution, while it is described as probabilistic peak expectation in line 118. There is also a value inconsistency in Table 1.

We really appreciate the reviewer’s detailed review and will carefully proofread the whole submission to fix the imprecise expressions and typos. Specifically, in the revision, we will modify line 118 to be the peak of probability and recalculate the Table 1.

Q1: First, I am not sure why did the author talks about number of input modalities as cross-modal retrieval and cross-modal imputation are not limited to one input modality. In fact, in most cases, like cross-modal imputation for sensory data, includes a large set of input modalities. Second, I think the author reaffirms my concern, which in the case that when there is no clear label information, the notion of inter-class geometry basically becomes not obvious (in the sense that n points will simply become n distinct classes).

A1: Thank you for the follow-up discussion. We will address the concerns one by one.

First, we really apologize about misunderstanding the modality number in cross-modal tasks. Now, we get a better understanding about the reviewer's meaning and agree with the possibility of conducting experiments of these tasks. However, due to time constraints, we have to admit that we cannot finish the experiments in time before ddl (Aug 13, AoE). We promise that we will report these experimental results of these tasks as soon as possible and include them into the revision. Here, we could only provide a brief overview of the implementation. If the task with labels, the implementation of PCD will be similar to that of the classification task. Without labels, it will resemble the implementation of the segmentation task.

Second, about the reviewer's concern, we would like to explain that it is not a problem, since the algorithm is by default compatible with the label-aware and label-free settings. There is a similar example regarding contrastive learning [1] and supervised contrastive learning [2] to help us clarify this. In constrastive learning, one sample is augmented into two views, and the representations of the two views are optimized to pull together and push far away from the representations of other samples. In supervised contrastive learning, the representations of samples from the same class are pulled together and pushed away from the representations of samples from the other classes. But generally, if the label information is available, the performance of supervised contrastive learning is better than the performance of contrastive learning, which to some extent aligns with the reviewer's opinion.

The difference between the contrastive learning-based $L_g$ in classification and segmentation tasks is analogous to that between supervised contrastive learning and contrastive learning. The former considers all $g\^\*$ sharing the same class as $g\_i$ as positive samples, whereas the latter uses $g\^\*\_i$ from the same instance as the positive sample. There is no concept for inter-class geometry in PCD, the calculation of geometric vectors $g_i,g\^{\star}_i$ is label independent and the significance of $L_g$ remains valid in segmentation, as described in A3.

[1] He K, Fan H, Wu Y, et al. Momentum contrast for unsupervised visual representation learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 9729-9738.

[2] Khosla P, Teterwak P, Wang C, et al. Supervised contrastive learning[J]. Advances in neural information processing systems, 2020, 33: 18661-18673.

Q3: I am confused by the response explaining L_g. The authors say that z* is not longer divided into positive and negative sets, but the loss function L_g is actually aligning the geometric vector g_i with the geometric vectors of other points in the positive group G_p right? Why does the author say that there is no positive and negative points anymore? Although, the new ablation study on L_g kind of addressed my concern, conditioned on the validity of the results.

A3: We apologize for the potential unclear description. Here, we will re-explain the computation procedure of $L\_g$ for the segmentation task based on its equation. Firstly, we list some important notations in the following table.

The term $L\_g$ aims to align $g_i$ with its single positive modality-complete counterpart $g\^{\star}_i$ in a contrastive learning-based algorithm, as expressed by the following equation:

$

L_g = - \mathrm{log} s(g^{\star}_i,g_i) = - \mathrm{log} \frac{\mathrm{exp}(\beta(g^{\star}_i,g_i)/\tau)}{\mathrm{exp}(\beta(g^{\star}_i,g_i)/\tau)+ \mathrm{exp}(\beta(g^{\star}_j,g_i)/\tau)},

$

where $\beta(g,g^{\star})$ calculates the cosine similarity between $g$ and $g^{\star}$, $\tau$ is the temperature coefficient. It is worth noting that in segmentation, due to the high dimensionality of multimodal features, only one negative vector $g^{\star}_j$ is selected to conserve computational resources (here is about the claim of positive and negative vectors in $L\_g$). Although the positive vector is only $g^{\star}_i$, $L_g$ still contributes to the conformal relationship between the peak points $\mu$ of $q(z|\mathrm{x})$ and the peak points $z\^*$ of $q(z|\mathrm{x})$ due to the alignment between $g^{\star}_i$ and $g_i$. We represent $g^{\star}$ by calculating the distances of $z^{\star}$, and $g$ is obtained by the distances of $\mu$, namely:

$

g^{\star}_i(b)=\alpha( z^{\star}_i, z^{\star}_b), g^{\star}_j(b)=\alpha( z^{\star}_j, z^{\star}_b), g_i(b)=\alpha(\mu_i,\mu_b),

$

where $g\_i, g\^{\star}_i, g\^{\star}_j$ are $|B|$-dimensional vectors with $\mu_i, z^{\star}_i, z^{\star}_j$ as the cores, respectively. $|B|$ is the batch size. Theoretically, $\alpha(\cdot, \cdot)$ can be any formula for calculating the distance between vectors. Notice that $g\_i, g\^{\star}\_i, g\^{\star}\_j$ are computed across all samples in the batch, without distinguishing between positive and negative points (here is about the claim of not requiring positive and negative $z\^\*$). The ablation studies on NYUv2 and Cityscapes in Table 7 in Appendix validate the effectiveness of $L\_g$ in segmentation tasks. Specifically, PCD includes all loss components outperforms the model with only $L\_c$ and $L\_u$ by an average of 1.37% and 0.86%, respectively.

We appreciate the reviewer's challenges about the clarity of this part, and will carefully improve this part with the notation table and more detailed explanation in the revision. All reviewer's advice will be incorporated to improve the submission.

Thank you for your time devoted to reviewing this paper and your constructive suggestions.

Q1: Some equations are misleading. For example, in Eq.(2), it is not clear that whether the authors mean the accumulative probability of $z\_p^\*$ is larger than $z\_n^\*$, or exactly mean each $z\_p^\*$ to $z\_n^\*$. Similar cases happen in Eq.(3). I think the authors should clarify this part due to their intrinsic difference and different implication. Besides, the similarity measure $s(\cdot, \cdot)$ cannot be any metric, since it occurs inside the log operation (Eq.(5)) and should be satisfy the positive constraint.

A1: Eq.(2) requires that the probability of any one positive point $z\_p^\*$ is larger than the probabilities of all negative points $z\_n^\*\in Z\_n$. Eq.(3) has a similar meaning, namely, for any $g^⋆\_p \in G\_p$, the inequality $s(g^⋆\_p, g\_i) \gg s(g^⋆\_n \in G\_n, g\_i)$ should be hold. And for $s(\cdot, \cdot)$, thanks for pointing out the areas where our presentation is unclear.

Q2: It is not clear why the Eq. (7) and Eq. (9) share the coefficient in Eq. (10). Is this optimization stable? Are they in the similar scale in terms of the loss range. At least, the authors should give some evidences or some ablations to show the rationality about the shared $\lambda$ in Eq. (10).

A2: The Eq.(7) and Eq.(9) are realizations of the Probability Extremum and Geometric Consistency components of Eq(5), respectively. And Eq.(5) is the simplification of the objective Eq.(4). In this simplification process, no distinct coefficients are generated for the two terms in Eq.(5). Thus we share the coefficients of Eq.(7) and Eq.(9) in Eq.(10). Besides, we also conduct experiments on CeFA to explore the impact of using different coefficients. The results are shown below. As can be seen, setting different hyperparameters $\lambda\_{u},\lambda\_{g}$ does not cause significant performance fluctuations.

Q3: It seems that the improvement of the experiments in Table 1 for NYUv2 and Cityscapes are smaller than one. It is better for these experiments with multi-round trials and report the std for the statistical signification explanation.

A3: In Table 6 in Appendix, we detail the stability experiments for PCD across four datasets. Each experiment is repeated for three times, allowing to calculate the average score along with the standard deviation. The results reveal that, even in its worst-case scenario, PCD outperforms the best competing methods. These outcomes not only underscore PCD’s superior performance but also attest to its stability and consistency across a wide range of segmentation testing conditions. In addition, to further confirm the effectiveness of PCD on segmentation tasks, we conduct experiments on a larger dataset, SUN RGB-D[1], which contains 5,285 RGB-Depth pairs for training and 5050 pairs for testing. The results are shown below. We can see that PCD is effective even on a larger segmented dataset.

I read the replies, which addressed most of my concerns. I would like to raise my score.

Dear Reviewer EqMG,

We sincerely appreciate you taking the time to review our responses and contributing to improving this paper. We will carefully follow the reviewer's advice to incorporate all the addressed points with additional exploration in the updated version. Thank you once again for your dedicated and valuable contribution in reviewing our paper!

Best,

Authors of Paper 14962.

Thank you for your time devoted to further comments. We would like to make more detailed explanations to figure out your concerns.

Q1: Notations are not clearly defined. Moreover, the writing shall be further improved. There is no general logic in formulating the methodology.

A1: We really appreciate the reviewer’s detailed review and will comprehensively polish the writing by clarifying the general logic, and carefully proofread the submission to fix the typos and grammatical errors.

Q2: A stronger motivation should be proposed to justify the proposed two properties. For example, maybe some empirical evidence that some missing modalities still follow a similar data distribution as the complete ones, then it would be reasonable to propose probability extremum property and further design the learning objective.

A2: Thanks for your suggestions on our motivation. We use t-SNE to visualize the distribution of the modality-complete, RGB, Depth, and IR representations of the unified model without PCD distillation. The results are shown in Figure 1 in the pdf. It can be observed that each unimodal distribution is similar to the modality-complete distribution, which provides empirical evidence for PCD to consider the indeterminacy in the mapping from incompleteness to completeness.

Q3: In the experiments, how did the authors control the missing rate of modalities? Moreover, the sampling strategy could be essential, how did the authors sample missing modalities? Is it randomly sampled or uniformly sampled?

A3: We are sorry for missing the information about the augment setting. In this paper, we conduct experiments under two modality-missing settings: 1) training with modality-complete data and testing with modality-missing data; 2) training and testing with the modality-missing data.

• Most of the experiments are under setting (1). During training, we augment each modality-complete sample by simulating all potential missing modality scenarios and randomly sample one of the augmented data as the training sample for the current epoch. Thus, the missing rate is $\frac{2^M-2}{2^M-1}$, where $M$ is the number of modalities, and samples with missing modalities are uncertain in each epoch. Furthermore, to investigate the impact of the random augmentation strategy, we conduct additional experiments with extreme augmentation conditions, namely, excluding simulations that only RGB, Depth, or IR modality is available. The results on CASIA-SURF are shown below. It can be seen that when the random augmentation is no longer applied, there is a decrease in performance. These results indicate that appropriately simulating and sampling various missing scenarios helps enhance the multimodal robustness. During testing, we build various testing sets with different missing cases, where each set contains only one missing case using the entire testing data. We report the results on each testing set, as well as the average results across them.

• The experiments in Table 5 in the main paper and Table 9 in Appendix are under setting (2). During training, we augment each data by simulating all possible modality-missing cases for this data. During testing, we build the same testing sets as setting (1). In Table 9, we evaluate the performance on both the CASIA-SURF and CeFA datasets, where each modality of the training data has either 30% or 40% of its data missing. As can be seen, the missing rate for training data could affect the final results under setting (2), where the missing rate is larger, the performance is worse. This is because that PCD is only applied to the data that has a modality-complete counterpart, a large missing rate means that less data is used for distillation.

Q4: What is the difference between learning under datasets with three modalities and two modalities? Does the change of modalities number affect the learning performance of the proposed method? Can the authors make some explanation with respect to this part?

A4: To answer the reviewer's question, in the following, we explore the impact of the number of modalities by controlling it on CASIA-SURF. The experiments are performed under setting (1) and the results are shown below. Notice that, the higher the number of modalities, the larger the missing rate of the training set. For example, the missing rate is 2/3 for two-modality and 6/7 for three-modality. From the results, it can be observed that with more modalities, better modality-complete representations can be provided, so as to transfer more privileged information. This results in a model that is relatively robust across various missing cases. When the number of modalities is small, there are fewer missing cases that need to be considered. The model may focus more on fitting an easy missing case, resulting in marginal improvements. For example, when complete modalities are RGB and Depth, the result of two-modality data at Depth (1.73%, lower is better) is better than that of three-modality (2.20%).

Dear Reviewer upEV,

Thank you for your constructive feedback and comments on our submission. We believe that these comments have significantly strengthened our work. We would greatly appreciate it if, upon reviewing our revisions, you could help champion our submission in the next phase or consider raising the score to support our submission given to the current diverged score ratings. Thank you once again for your thoughtful feedback and for considering our request.

Sincerely,

The authors of 14962