Generalizable Cross-Modality Distillation with Contrastive Learning

We thank the reviewer for the thoughtful assessment and for the positive evaluation of our work. We address the points raised in the review below.

W1: A comprehensive comparison with recent works on multi-modal distillation

Response: Thanks a lot for the suggestion. We will explain the difference between our method and these two works and then elaborate on the difficulty of using these two methods in our problem. Furthermore, we conducted an experiment on the momentum distillation technique in both works to show that it does not give an obvious improvement in the performance but introduces a computational burden.

The ALBEF[1] introduces a contrastive loss to align image and text features, which enables more grounded vision and language representation learning. Besides, they propose momentum distillation to improve learning from noisy web data. The alignment process in the ALBEF is to make the image and text features more grounded so that the features can be used for further cross-modal applications. However, in our problem, we want to distill the knowledge from a 'rich' modality to a 'limited' modality and aim at boosting the model performance only in the target 'limited' modality. Since the limited number of training data in the target modality, it may be difficult to construct a general feature space for both modalities. As the second contribution of ALBEF, the momentum distillation is designed to leverage a larger uncurated web dataset which is actually unavailable in our problem since the data in the target modality is limited. The key idea of our work is to fully investigate the relationship behind the paired data while the idea of momentum distillation is to eliminate the influence of noisy data which are not so compatible. In conclusion, the ALBEF is mainly designed for image-text modalities with large noisy paried data and its modules are not so feasible for the task we study.

The COMMON[2] is developed for the graph matching problem which introduces quadratic contrastive loss consisting of traditional infonce loss, within-graph consistency loss, and cross-graph consistency loss. Another technique momentum distillation is also designed for robust matching which aims at eliminating the noisy label in a large dataset. In our problem, the number of available data is limited which can be only 700 in image-depthmap scenario in which case such a momentum update may not introduce obvious benefits in performance but additional computation cost.

We added the experiment in the Appendix D.4 in the revised version. We test the momentum distillation and within-regularizer techniques with our method. M-CMC indicates the CMC loss combined with the momentum update. In Table 10, we show that there is not an obvious improvement when we use the momentum update in the distillation process. Specifically, when applying the momentum update to image-sketch modalities it helps the distillation to some extent but for the other two, there is no improvement. Furthermore, we try the within-regularizer in the target modality when distillation, denoted as WR. In some cases, the WR actually helps the distillation like VGGSound on LE, but gives no improvement or a similar result on other tasks. The experiment results just fit with the arguments we made above and show the difficulty of applying such techniques to our problem.

We thank the reviewer for the thoughtful assessment and for the positive evaluation of our work. We address the points raised in the review below.

W1: The proposed method follows the self-supervised knowledge distillation framework which is widely used in existing works

Response: Compared to the existing methods of multi-modal learning or cross-modality transfer, e.g., [1], our framework has the following novelty and significance.

(1). Our framework stands out for its generalizability in multiple ways. Unlike the framework in [1], which focuses on fixed modalities like image-text and requires specific modifications for each modality, our framework demonstrates its ability to work with various modalities, as supported by our experimental results. Moreover, our method allows the distilled feature space from the source modality to be adapted to diverse downstream tasks, as evidenced by our results.

(2). Effective utilization of paired and unlabeled data. Another notable distinction between our framework and [1] is the effective utilization of data. Our method is designed to distill information from a 'rich' modality to a 'limited' modality using only a small number of paired data, whereas [1] relies on a large number of paired data for distillation or learning general pattern space.

(3). Theoretical foundation. Our method is supported by a rigorous theoretical convergence analysis, which is a key contribution of our work. To the best of our knowledge, no previous work has established theoretical results for the convergence or boundedness of contrastive learning-based distillation frameworks like ours.

W2: However, it seems that the negative correspondence is only used in the pretrain stage of source modality.

Response: Compared to the losses like the CMKD which only minimizes the feature distance between the positive paired data between source and target modalities. The cross-modality contrastive loss will minimize the distance between positive paired data and maximize the distance between the negative paired data. Specifically, in the equation

$-\log \frac{\exp(z_i^{\mathcal{A}} \cdot z_i^{\mathcal{B}}/\tau)}{\sum_{t} \exp(z_t^{\mathcal{A}} \cdot z_i^{\mathcal{B}}/\tau)}$

$z_i^{\mathcal{A}}$ and $z_i^{\mathcal{B}}$ are positive paired data and $z_i^{\mathcal{A}}$ and $z_j^{\mathcal{B}}$ ($j\neq i$) are negative paired data. When minimizing the loss, we will expect $-z_i^{\mathcal{A}} \cdot z_i^{\mathcal{B}}$ to be small and $-\sum_{j\neq i} \exp(z_i^{\mathcal{A}} \cdot z_j^{\mathcal{B}})$ to be large in which case will minimizing the cosine distance of positive data but maximizing the cosine distance of negative data. In this way, we can fully leverage the positive and negative relationships or alignment between source and target modalities.

W3: There are too few ablation experiments in the paper.

Response: Thanks for your suggestion. We add more ablation experiments and discussions in Appendix D in the revised draft. For the transformer structure, we added experiments on ViT-S/16 and ViT-B/16 to show the effectiveness of our method. From the results in Table 9, we can find the base performances of ViT are lower than ResNet with a small number of training data while our method still helps the learning in the target modality with about 4-7% improvement.

Q1: Why not utilize unpaired data in the target modality?

Response: In the experiments of images and sketches, the paired data and the finetuned data are disjoint as described in the experiments setup in Section 4 and detailed experiments setting in Appendix. C. The main idea of our framework is that we can learn a generalizable representation from a well-studied modality and the contrastive distillation with paired data. So based on the learned feature, we can still use the unpaired data in the target modality.

Q2: If there is no labeled data in the target domain, is the model still effective?

Response: Actually, our framework is primarily designed to learn a better and more generalizable representation/feature space in the target modality by using cross-modality contrastive distillation. And in this case, it does not need labeled data in the target domain. In experiments, we demonstrate this point by only training a linear layer upon the learned feature extractor and decreasing the number of labeled data as described in the last ablation study in Section 4.2. When only using 1-shot/5-shot labeled data in the target domain our method can achieve a similar performance with the supervised finetune.

Q3: In Table 1, what’s the downstream task of each dataset? It’s not clear to me in its current form.

Response: Thanks for the suggestion, we have described the downstream tasks in the Experiments Setup at the beginning of Section 4 Experiments and Discussion. To make it more clear, we have added the downstream tasks information in the Table 1 in the revised version.

We thank the reviewer for the thoughtful assessment and for the positive evaluation of our work. We address the points raised in the review below.

W1: The motivation for the proposed distillation is not clear

Response: The motivation or reason why our pipeline can work is that the number of data in the source modality can be much larger than the target modality which will lead to 'rich' knowledge in the source modality and 'limited' knowledge in the target modality as we mentioned in the introduction. From this view, the pipeline of SSL(source) + alignment(source + target ) can distill the 'rich' information from the SSL(source) via the alignment to the target modality which will be better to only leverage the 'limited' knowledge in the target modality, i.e., SSL(target) + FT(target). Specifically, in our experiments, we pre-trained the SSL model on ImageNet with more than 1 million images but only 60k sketches available in the target modality.

W2:Theoretical analysis cannot support why the losses are useful

Response: To the best of our knowledge, this is the first work to establish a theoretical convergence analysis of such a contrastive learning-based distillation framework which is a key contribution of our work. Moreover, there are no theoretical results of the convergence or boundness of the previous methods such that we can not compare the theoretical performance of our method and previous methods.

W3: I would like to see a more detailed analysis of why this happens

Response: We have explained this problem in the first response and want to provide another analysis of our method and why our method works. In simple words, SSL (source) - alignment (source + target) will provide a better pre-trained start point than SSL (target) - FT (target) or FT(target) since we leverage the much larger number of training data in source modality than the target modality. If we assume that good performance on the target(e.g., high-quality sketch) downstream tasks, e.g., classification is closely related to the goodness of the representation we learn in the target modality, then given a small number of training data in the target modality, it will be difficult to learn a comprehensive representation. However, there may exist a source modality (e.g., image w.r.t high-quality sketch) with a large number of available data. By leveraging enough training data and SSL in the source modality, we can get a well-learned and powerful representation space of the source modality. Alignment in our work will distill the information from the learned representation space to the target modality and help to learn a better representation space in the target modality. In this way, we will get a better pre-trained model for the tasks in the target modality.

W4: Experimental details are missing

Response: In fact, we have provided information like epochs/lr of SSL, alignment, and FT in Appendix.C on page 19. Besides, we also provide the code for reproducibility.

We thank the reviewer for the thoughtful assessment and for the positive evaluation of our work. We address the points raised in the review below.

W1:The novelty of the method itself is rather limited

Response: Though the losses used for the cross-modality distillation are an adaption of self-supervised distillation and CLIP, our method is designed for a totally different task which is a provable and more generalizable framework. One of the key differences between our framework and self-supervised distillation/CLIP is that our method is designed to distill information from a 'rich' modality to a 'limited' modality via only a small number of paired data, but both self-supervised distillation and CLIP needs a large number of paired data to distill or learn general pattern space. In another way, self-supervised distillation only works for single-modality situations and CLIP is mainly designed for image language while our framework can work for any modalities only when there is some paired relationship that can be investigated. Besides, we give the theoretical analysis to further understand the algorithm which is also a key contribution of the work.

W2:The analysis on the relationship between the performance and estimated total variation distance between two datasets.

Response: Thanks for the suggestion. We add new experiments and discussions in Appendix D.1 in the revised draft. To provide object evidence, we compute the estimated total variation distances and performance improvements of several datasets and tasks, specifically, we use the performance difference between CMD + LE and SSL + LE as LE($\Delta$), CMD + FT and Sup FT as FT($\Delta$). As shown in Table 7, our discussion in Section 4.1 actually fits with the estimated TV distance that image-sketch has a smaller modality gap than video-audio. The performance improvements of image-sketch are more significant than video-audio from the results in Table 7. Since the downstream task of image-depthmap datasets is segmentation which is different from the classification task as image-sketch and video-audio, it is unfair to compare the results directly with other two cases. But in general, the quantitative results we provide here conform to the discussion in Section 4.1.

W3:What are the values of M and m?

Response: The values of $M$ and $m$ mean the number of the whole paired data and the number used in the distillation respectively. For example, in the ablation study, we may have a whole number of $M=15000$ paired image and sketch while we only use $m=3000$ in the distillation and then the $m/M=20\%$ means that we use 20% data for distillation. We have added the explanation of $M$ and $m$ in the discussion and given the true values in detailed experiment settings in the appendix.

W4:Typos Thanks a lot for the suggestion, we have fixed all typos in the updated version.

Q1:Is it possible to apply both CMD and CMC?

We add new experiments and discussions in Appendix D.2 in the revised draft. Absolutely, we can combine these two losses when distilling, we further conduct an ablation study to investigate how the combination affects the final performance. From Table 8, we can find that combining CMC and CMD losses may help the distillation on some tasks, but there is no dominant choice for every task. Theoretically, I think the combination will make the analysis much more difficult without any more assumptions. We think this is beyond the contribution of the work and can be investigated in future work.