Extending Multi-modal Contrastive Representations

W1: grey mark in Table 1 and Table 2:

A1: CLIP and CLAP are used as base-MCR and leaf-MCR, and they only focus on contrastive representation between two modalities (i.e., image-text and audio-text). Our experiments mainly focus on comparing the unified representations for three or more modalities over various cross-modal downstream tasks rather than the performance on specific image-text or audio-text tasks. The results of CLIP and CLAP are listed to compare how well C-MCR and Ex-MCR inherit the semantic alignment of original MCR spaces. Moreover, in the responses to reviewer 6GyA’s W1 & Q1 & Q3, we discuss in detail why Ex-MCR’s performance on audio-text or 3D-image lags behind that of the leaf-MCR is acceptable and some future directions to further improve performance.

In Table 1, the baseline WAV2CLIP is trained on VGG-Sound, and the evaluation dataset VGGSS is a subset of the VGG-Sound test set. This means the evaluation of WAV2CLIP on VGGSS is supervised. However, other methods do not use any VGG-sound data for training, so other results on VGGSS are evaluated in a zero-shot manner. Therefore, we grayed out the results of WAV2CLIP on VGGSS.

Sorry for the misunderstanding caused by not clearly explaining the gray markings, and we have added these explanations in the newly submitted version.

W2: 11.30 and 11.19 in Table 3

A2: I suppose you are probably referring to 11.30 and 11.19 in Table 4. Sorry for the misunderstanding, this is a typo, 11.30 should be bolded, not 11.19. We are not intentionally bolding it incorrectly. In the similar case in Table 6, the marks are correct.

Moreover, in Table 11, we display more ablation results about alignment objectives. Using all alignment objectives shows obvious advantages over most tasks and metrics. This incorrect bold mark would not affect the conclusion that the dense alignment objective is effective.

Thank you for your kind reminder. We have modified this typo in the newly submitted paper.

W3 Considering the issues raised earlier, the reviewer remains uncertain about the authenticity and reproducibility of the results.

A3: The gray marks in Tables 1 and 2 are explained in the response to W1. For W2, it is a typo and it has been modified. The detailed ablation experimental results are also shown in the appendix, further supporting our conclusions.

To completely eliminate concerns about the authenticity and reproducibility of the results, we provide Ex-MCR pre-trained weights, well-packaged code, download links for the pre-trained models used, and guidelines for using our Ex-MCR in the new submission's supplementary materials for reproduction. These pre-trained weights and code will be open source.

W4 Additional findings and conclusions drawn from the ablation studies?

For ablation results on “Various modality-centric data”, we further find that pseudo-pairs from audios are critical to the performance of audio-text retrieval, demonstrating the importance of various modality-centric data, and proving that previous single modality-centric data really can not fully reflect the audio representation space.

For ablation results on “Dense alignment objective”, we find that directly aligning the pseudo audio-image or audio-text embedding pairs leads to sub-optimal audio alignment, whereas aligning spaces by overlapping text modality brings better alignment than learning alignment directly from pseudo pairs. This observation further suggests that overlapping modalities play a key pivotal role in aligning different spaces.

For ablation results on “Structure of $f_m(\cdot)$”, we summarize that good results are achieved no matter how many layers of MLP, which demonstrates the robustness of our method. According to more detailed experiments, empirically, MLP with 2 or 3 layers achieves a good balance between expressivity and learning difficulty.

Dear Reviewer GFCr,

Thanks again for your comments. We would like to kindly remind you that we tried our best to address the concerns you raised.

We kindly request your feedback as the rebuttal deadline is approaching in less than 1 day. We would be happy to discuss in detail if you have additional comments about our paper.

Best regards, Authors

Q3: The explanation of $\{\tilde{\mathbf{a}}_i^A, \tilde{\mathbf{t}}_i^A, \tilde{\mathbf{t}}_i^I, \tilde{\mathbf{v} }_i^I\}$

Since the pseudo data pairs retrieved using different modalities are shuffled to form the final data pool, we do not distinguish symbolically between data pairs from different sources. The $\{ \tilde{\mathbf{a}}_i^A, \tilde{\mathbf{t}}_i^A, \tilde{\mathbf{t}}_i^I, \tilde{\mathbf{v} }_i^I \}$ represents a pseudo-pair sample in the final data pool, and this sample can be generated from any source.

Thank you very much for your appreciation of our work and your valuable suggestions on the presentation of our manuscript. In our newly submitted PDF, we have carefully revised the presentation of most paragraphs to improve its clarity and readability. We hope the new submission can solve your concerns about writing.

W1: statements about ULIP

ULIP aligns a 3D encoder to an image-text model called SLIP [1] (not CLIP) through 3D-image-text data. Ex-MCR only uses the aligned 3D-image representation of ULIP and extends it to a different image-text model (i.e., CLIP) via the paired-data-free way. So we are not reproducing or refining the 3D-text alignment of ULIP, but building a new alignment between the 3D representation and new image-text pre-training.

[1] Slip: Self-supervision meets language image pre-training. Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. ECCV 2022

W2: repetition of acronyms

Thanks for your suggestions, in the newly submitted paper we carefully refine the usage of acronyms to improve readability. The modifications of acronyms can be summarized as:

1. Avoiding confusion due to many similar acronyms. Replace “MCR” with "space", "intra-MCR alignment" with "intra-space alignment", and "inter-MCR alignment" with"inter-space alignment". Remove unnecessary acronyms, like “MLP”, “InfoNCE Loss” and “L2 Loss”
2. Use "audio-text space" "image-text space" and "image-3D space" in the introduction instead of using "CLAP", "CLIP", and "ULIP" to directly explain the aligned modalities in each space.

W3 & W4: Statements in the “modality-centric consistency” part

We reorganize and rewrite the entire "modality-centric consistency" part in the new submission.

In the introduction, we avoid directly giving many self-defined terms without explicit explanation, such as: “modality-centric”, “semantic bias” and “reflect an MCR space comprehensively”. The so-called modality $\mathcal{A}$-centric pseudo data means using only data of modality $\mathcal{A}$ to retrieve semantically similar data in other modalities, while "semantic bias" means that one modality cannot be fully represented by another modality (giving examples of audio and images to support), and "various modality-centric data" denotes combining the pseudo pairs retrieved by different modalities.

In the first and second paragraphs of the Section 3.2.1, we define the "single modality-centric data" and "various modality-centric data", and provide examples of image of “mushroom” and audio of “wind noise” to illustrate that single modality-centric data cannot comprehensively reflect representations of different modalities in different MCR spaces.

W5 & Q1 & Q2: unclear notation

The $softmax(\cdot)$ in Equation 1 and Equation 2 is over all the samples in used datasets.

The InfoNCE function in Equation 6 is calculated on all the samples in a training batch.

The tilde symbols mean the features are processed to be semantically consistent.

W6: The metrics in Table 3,4,5,6

At the caption of Table 3 and the beginning of Section 4.5, we add descriptions about the metrics used in Tables 3, 4, 5, and 6. In ablation experiments, we report the mAP metrics on audio-image retrieval (AVE) and audio-text retrieval (AudioCaps). In the Appendix, we provide detailed ablation experiment results on all the datasets and metrics.

W7: Too crowded Figure 1

We redesign Figure 1 and rewrite its caption. Figure 1 is divided into two sub-figures (a) (b).

To improve the simplicity of the figure, we made the following changes to Figure 1 (a):

1). Remove unnecessary text such as "pull close" and "pull close & push away" and replace them with a gray bold dash line.

2). Replace text such as "audio features" "CLAP Text features" with corresponding vector symbols. On the one hand, keeping there are only vector symbols, modules, and lines in the pipeline figure, can display the calculation process more clearly. On the other hand, it echoes the notation definitions in Section 3.2 and helps the reader to quickly understand the meaning of the defined notations.

3). Frame the base-MCR and two leaf-MCRs into three boxes, which not only improves the aesthetics and simplicity, but also emphasizes that the three pre-trained spaces are separate.

4). Add explanations about some symbols at the bottom to help understand the illustrations.

For Figure 1 (b), the subfigure about various modality-centric data

we cancel the repeated "text-centric consistency", "audio-centric consistency", and "image-centric consistency" which are difficult to understand. More emphasis is placed on combining pseudo-data pairs from different sources to form a data pool.

For captions, we do not broadly explain the entire picture, but provide relevant explanations for each sub-picture. And use more space to explain the more complex Figure 1 (b).

Dear Reviewer pnPG,

Thanks again for your comments. We would like to kindly remind you that we tried our best to address the concerns you raised.

We kindly request your feedback as the rebuttal deadline is approaching in less than 1 day. We would be happy to discuss in detail if you have additional comments about our paper.

Best regards, Authors

W1: the imagebind baselines & the limitation/boundary of this kind of paired-data-free methods

Comparison with imagebind:

The current compared baselines: AudioCLIP, WAV2CLIP and ULIP, ULIP v2. These methods are all trained on paired audio-image-text or 3D-image-text data, and their model sizes are equivalent to Ex-MCR. Thus, the main difference between Ex-MCR and these baselines is the training data and training strategy. Comparison with these baselines better demonstrates the strengths of our “extending” learning scheme.

However, imagebind not only uses paired data for training, but also uses a much stronger pre-trained model, (Imagebind uses OpenCLIP ViT-Huge trained on LAION-2B while Ex-MCR uses CLIP ViT-Base trained on WIT-400M). Considering the audio-image-text representation, the parameters number of imagebind are 630M (image encoder), 303M (text encoder), and 86M (audio encoder), while the parameters number of our Ex-MCR is 88M (image encoder), 63M (text encoder) and 32M (audio encoder).

Although imagebind uses both data pairs and stronger pre-trained models, we still provide a performance comparison on audio-image-text as:

On the image-audio task, imagebind achieves better results since it uses large-scale paired data and stronger visual pre-training. On the audio-text task, our method is far better than imagebind. The performance difference in image-text tasks is due to using the different CLIP models.

The potential of this kind of paired-data-free methods

In the responses to Reviewer 6GyA, we fine-tune the projector with paired data which brings performance improvements. Besides, we further discuss the potential of Ex-MCR in scalability and low computing resource situations.

Regarding the boundaries and potential of absolute performance, we try to replace the CLIP ViT-Base with the larger OpenCLIP ViT-Huge. The results are also reported in the above table. Simply using a larger version of CLIP, the performances on audio-image, audio-text, and image-text tasks are significantly improved.

In addition, there are many future directions to further improve the absolute performance of Ex-MCR, such as using more unimodal data, integrating paired data into the training process, employing stronger leaf-MCR pre-training, and designing more sophisticated space alignment architecture and learning objectives. Our future work will be dedicated to further exploring the potential of such paired-data-free methods.

Q1: Typos. Unclear Figure 1 and its caption.

Thanks for pointing out the typos and providing suggestions about figures, we have fixed them in the newly submitted paper.

Additionally, in our response to reviewer pnPG, we analyzed most of the modifications that were made to improve the clarity and readability of the presentation.

Dear Reviewer CZb4,

Thanks again for your comments. We would like to kindly remind you that we tried our best to address the concerns you raised.

We kindly request your feedback as the rebuttal deadline is approaching in less than 1 day. We would be happy to discuss in detail if you have additional comments about our paper.

Best regards, Authors

W2 & Q2 & Q4: The new MCR should have a modality already present in the base MCR for effective expansion. And the results of extending new MCR via leaf-MCR.

As discussed in recent work imagebind [1] and languagebind [2], most existing modalities can be bound to either images or language, and most existing MCRs contain either images or language. From a practical perspective, choosing the powerful CLIP as base-MCR can cover almost all scenarios.

[1] ImageBind: One Embedding Space To Bind Them All. Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, Ishan Misra. CVPR2023

[2] LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment. Bin Zhu, Bin Lin, Munan Ning, Yang Yan, JiaXi Cui, Hongfa Wang, et al. Arxiv2023

However, to further explore the potential of Ex-MCR, we still try to expand a leaf-MCR through the overlapping modality of another leaf-MCR, like a chain structure. As you mentioned in Q4, we regard CLAP as base-MCR and CLIP as leaf-MCR, the audio-image-text results are：

When using CLAP as base-MCR, the audio-text retrieval accuracy on AudioCaps is much higher than using CLIP as base-MCR. Correspondingly, its image-text retrieval accuracy on COCO is lower than using CLIP as base-MCR. For audio image retrieval, the performance of both variants is comparable.

The most important experiment: use ULIP as leaf-MCR of leaf-MCR, which means indirectly aligning the 3D representation of ULIP to the text representation of CLAP through CLIP. The final 3D-image-text results are:

The final 3D-text-image performance is indeed lower than directly using ULIP as a leaf-MCR and using CLIP as base-MCR. But it still achieved performance comparable to most baselines, which shows the error accumulation of chain structure is within the acceptable range.

Q5: Training resources:

In our implementation, extending CLAP into CLIP only requires 4G GPU memory. When training on an A100, it takes 6 hours to converge. Due to its extremely low GPU memory requirements, users can train Ex-MCR on almost any device. We also try to reduce the batch size, and when only 2GB GPU memory is used (of course requiring a longer training time), there is almost no performance degradation.

Our training is extremely resource-efficient. Previous contrastive representation learning methods need to set a larger batch size for training stability, which results in very high GPU memory usage. However, in our method, all pre-trained encoders are frozen, so all features can be pre-extracted and saved offline, and only a linear and MLP layer-based projector is learnable, which greatly reduces the GPU memory cost when increasing the batch size.

W1 & Q1 & Q3: why Ex-MCR’s performance lags behind that of the leaf-MCR and possible solutions.

1) Taking extending CLAP to CLIP as an example, if you mean why the projected CLAP's audio-text representations lag behind the original CLAP audio-text representations:

In order to avoid misunderstanding, we further clarify that the final unified Ex-MCR representations are composed of CLIP's image, text representations, projected CLAP's audio representations, and projected ULIP's 3D representations (i.e., the $\mathbf{t}^I_i$, $\mathbf{v}^I_i$, $f_m^A(f_l^A(\mathbf{a}^A_i))$, $f_m^U(f_l^U(\mathbf{p}^U_i))$ are the final audio-text-image-3D unified representation). All our experimental results are based on these four unified aligned representations. Therefore, the audio-text results are evaluated by the projected CLAP's audio and CLIP text representations (i.e., the $\mathbf{t}^I_i$, $f_m^A(f_l^A(\mathbf{a}^A_i))$ ), rather than the projected CLAP's audio-text representations (i.e., the $f_m^A(\mathbf{t}^A_i)$, $f_m^A(f_l^A(\mathbf{a}^A_i))$ ).

2) Taking extending CLAP to CLIP as an example, if you mean why the newly aligned audio-text representation in Ex-MCR's unified space lags behind the CLAP audio-text representations:

The following are reasons and possible solutions for performance lags behind leaf-MCR.

Reason for performance lags behind leaf-MCR:

We think that the performance of Ex-MCR lags behind that of leaf-MCRs for three reasons:

1、Unchanged features extraction capability. The final Ex-MCR's audio representation is only a transformation of the CLAP's audio representation, and the ability to extract meaningful features from the original data is unchanged.

2、Non-optimal target space. The representation distribution in fixed base-MCR may not fully distinguish the modalities of leaf-MCRs. For example, “people are talking on the beach” and “people are talking in the living room” are different in CLIP’s text representation, but are similar in the audio-text domain.

3、Limited training data. The pseudo data we use to extend leaf-MCR to base-MCR are mainly derived from the inherent alignment of leaf-MCR and base-MCR. Therefore, this extending process can be regarded as distilling leaf-MCR’s alignment knowledge to construct a new alignment for base-MCR.

Moreover, Ex-MCR is designed to learn unified representations for more than three modalities rather than for only two modalities. And current experiments in the paper aim to demonstrate the potential of Ex-MCR in the extreme situation of no data pairs can be used. Considering the above facts, we think it is reasonable and acceptable that the current Ex-MCR lags behind leaf-MCR in performance on downstream tasks in leaf-MCR’s specific modality.

Possible solutions for performance lags behind leaf-MCR:

Combining the above analyses of limitations, there are many possible solutions to further improve the performance of Ex-MCR. Such as using more unimodal data, integrating paired data used by leaf-MCR into the training process, and employing stronger leaf-MCR and base-MCR pre-trainings.

As you suggested in Q3, we try to integrate the paired data used by leaf-MCR. With limited rebuttal time, we simply use part of CLAP’s training audio-text data (audiocaps training set, and 200k pairs of laion630k) to fine-tune the projector. The results are as follows:

By tuning the learned projector with audio-text data pairs, the performance of audio-text is greatly improved without sacrificing the audio-image performance. This demonstrates the compatibility of our structure with paired data, and paired data can further boost the final performance.

We also evaluate using a larger base-MCR pre-training, which also brings performance improvement. Please refer to the response to Reviewer CZb4's W1.

Dear Reviewer 6GyA,

Thanks again for your comments. We would like to kindly remind you that we tried our best to address the concerns you raised.

We kindly request your feedback as the rebuttal deadline is approaching in less than 1 day. We would be happy to discuss in detail if you have additional comments about our paper.

Best regards, Authors