W1: Statements and Figure in the “Various Modality-centric Data” part

Thanks for pointing out the typos and providing suggestions about figures, we have fixed them in the new version.

1.1 Unclear Figure 1(b) and its caption.

We have uploaded the detailed captioned Figure 1(b) to the newly submitted PDF.

• We replace "softmax" with "A->B similarity" in the figure to emphasize that when using Audio or Image as the query, the shared element in the text aggregation process is the similarity weight results, not the softmax operation itself.
• We add a more detailed caption to better illustrate the overall pipeline of pseudo pair construction.

1.2 Typos in Eq(1)

We sincerely apologize for the misunderstanding caused by the typo. The correct formula (1) is as follows:

Eq(2) is correct. When audios serve as queries to construct pairs, the first step is to obtain the relevant text embeddings through the inter-modality alignment of CLAP. Note that $\tilde{\mathbf{t}}_i^A$ and $\tilde{\mathbf{t}}_i^I$ always share the similarity weights retrieved by CLAP (the softmax term in the formula) during data aggregating. Next, we use $\tilde{\mathbf{t}}_i^I$ and the inter-modality alignment in CLIP to obtain the relevant $\tilde{\mathbf{v}}_i^I$.

W2: Explanations for performance enhancement on inter-modalities when compared to C-MCR

To avoid misunderstanding, we further clarify our experiment settings in the response to all reviewers(Part 1). Benefit from the fully enhanced alignment learning pipeline described in Section 3.2, Ex-MCR makes significant progress in preserving existing inter-modalities alignments in leaf space(Audio-Text and 3D-Image) and has steadily improved the newly learned inter-modalities(Audio-Image and 3D-Text) alignments, compared to C-MCR.

Most importantly, Ex-MCR fully inherits the inter-modalities alignments in base space, significantly outperforming C-MCR. The fully preserved base space brings strong modality scalability. Serving image or text as overlapping modalities, various multi-modal contrastive learning spaces can be projected into a unified Ex-MCR space.

W3: the analysis of hyperparameter choosing

Please refer to the response to all reviewers(Part 3).

Q1: Inconsistent wording

Thanks for pointing out the inconsistencies in wording. "audio-vision" and "audio-image" have the same meaning in the paper, and we have modified these two expressions into "audio-image".

Q2: trained on paired modality data

We finetuned Ex-MCR-huge on paired Audio-Text data (trainset of Audiocaps). The results are as follows:

Using paired data in training not only improves the alignment performance between the corresponding two modalities(Audio-Text) but also helps the model to obtain better alignment between other modalities(Audio-Image).

Thank you for the response, which addressed many of my initial concerns.

To further clarify my previous question, I believe it is crucial to present the oracle performance to validate the uni-modal training scheme. Specifically, how much does the model perform when trained on paired audio-image datasets? In addition, how much does the proposed Ex-MCR model benefit from a small amount of such paired audio-image training data after fine-tuning?

Thank you for presenting the additional experiments.

Despite using audio-image paired datasets for training, what could be the reason for the low performance (R@5) in audio-image retrieval tasks such as Flickr and AVE? Could you provide any insights into the reasons behind this?

Thank you for the additional experiments and further clarifications.

I believe the clarity of the paper has improved based on the authors' responses. I will increase the score to 5.

W1 & Q1 & Q2: Evidence for Information Loss & Support for Modality Alignment Forgetting

Taking the reprojecting CLIP representations to C-MCR space as an example, the following table(as a sub-table of Table 1) shows the performance differences before and after the projection:

In the reprojected CLIP representations, the Image-Text retrieval performance is significantly reduced, which indicates that the modality alignment of the original CLIP space is forgotten during the reprojection.

W2 & Q3: Comparison with Non-Projection-Based Methods

We compared Ex-MCR with non-projection-based methods, LanguageBind[1], UniBind[2], and PointBind[3] (Note that PointBind has the same models in Audio-Image-Text with ImageBind). For Audio-Image-Text experiments, we conduct the comparison under the same Image-Text performance.

The results are as follows:

Results of Audio-Image-Text experiments. The best results are bolded.

Ex-MCR achieved leading performance in 4 of the 6 Audio-Image retrieval metrics and a significant advantage in Audio-Text semantic alignment.

In 3D-Image-Text experiments, to explore the upper bound of performance in advanced pre-trained space, we use Ex-MCR to fuse EVA-CLIP-18B(Image-Text) and Uni3D-G(3D-Image-Text)(Please refer to the response to all reviewers). The newly constructed 3D-Image-Text space achieves state-of-the-art performance in all three domains (3D-Image, 3D-Text, Imge-Text), compared with non-projection-based methods.

Moreover, since Ex-MCR is extremely flexible, its performance can easily boost, and benefit from the breakthrough of the advanced pre-training spaces.

[1] LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment, ICLR 2024

[2] UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All, CVPR 2024

[3] Point-Bind & Point-LLM: Aligning 3D with Multi-modality, arxiv 2023

W3 & Q4: Validation of Contribution Claims

(1) Claim about preserving modality alignment:

Ex-MCR preserving modal alignment in two ways:

On the one hand, by extending the modalities in one space to another space rather than constructing a new representation space, Ex-MCR preserves the base space. For example, all Image-Text modality alignment of CLIP is fully inherited in Ex-MCR.

On the other hand, by redesigning the data aggregation method, the structure of the projector, and loss functions, Ex-MCR can preserve better cross-model alignment from the leaf space(Audio-Text and 3D-Image alignment, compared to C-MCR).

(2) Claim about integrating more pre-trained spaces:

In this unified space, we preserve advanced intermodal alignment (see Table 1, Table 2 in the paper), and obtain emergent 3D-Audio alignment (see Figure 2 in the paper and more diverse examples provided in PDF). This result proves that Ex-MCR is a unified representation space that can accommodate multiple modalities.

Moreover, the result also indicates that both image and text(modalities in base space) can serve as the overlapped modality to extend the space.

W4 & Q5: Differences between decoupled projector and existing projector

The decoupled projector reduces interference among different optimization objectives, which helps the model learn modality alignment better.

We performed ablation experiments on Ex-MCR-huge, and the results are as follows:

The above results show that the decoupled projector reaches overall performance.

W5 & Q6: Inclusion of Related Works

Thank you for providing more related works. We will cite these methods in the revised paper. Please refer to the response to W2 & Q3 for comparison results.

W1: the analysis of hyperparameter choosing

Please refer to the response to all reviewers(part 3 The analysis of hyperparameter choosing).

W2: 3D-Image-Text performance improvements are relatively limited.

For 3D-Image-Text experiments, we think there the main reason for the performance loss is the limited training data. In our latest experiments(referring to the official comment to all reviewers), we use Ex-MCR to fuse more powerful models, EVA-CLIP-18B(Image-Text), and Uni3D-G(3D-Image-Text). Ex-MCR achieves leading performance in all three domains (3D-Image, 3D-Text, Image-Text).

For the explanation of the Ex-MCR-base's 3D performance, please refer to the response to review nY4o's Q2.

W3：Ex-MCR requires additional training cost and inference time compared to C-MCR

Taking extending CLAP to CLIP as an example

For training, both Ex-MCR and C-MCR can be trained for less than 4 hours on a single RTX 4090, and the cost difference in reaching convergence can be ignored.

For inferencing, Ex-MCR only projects a single modality in one space, while C-MCR needs to project three modalities in two spaces, therefore C-MCR has a higher inference overhead than Ex-MCR. Detailed statistics for inferencing overhead are shown in the following table:

Params for projecting（M）

Computation overhead for projecting（GFlops）

W1 & W2: Lack of novelty in the article & the method is too simple

The main contribution of Ex-MCR is not to "apply contrast learning in multimodal retrieval tasks", but to efficiently integrate multiple pre-trained contrastive multimodal representation spaces into a new and more comprehensive one.

As a novel learning paradigm, Ex-MCR is extremely flexible and able to integrate multiple pre-trained contrastive learning spaces with different modalities to achieve state-of-the-art performance.

Moreover, the paper also redesigned the data aggregation methods, the structure of the projector, and the alignment objective function, all of which make Ex-MCR more powerful. We illustrate the motivation of these designs in Section 3.2 Enhancing Alignment Learning Pipeline. Our ablation studies in Section 4.5 Enhancing Alignment Learning Pipeline fully demonstrate the effectiveness of these novel designs. In the response to reviewer 1W4Z's W4&Q5, we further show the superiority of the decoupled projectors.

Q1：Why does not mark the 3D-audio retrieval results as correct or wrong

There is no existing benchmark for Audio-3D retrieval, so there is no objective definition of right or wrong for Audio-3D pairs. To demonstrate the emergent semantic alignment in Audio-3D, we directly show some top 5 retrieval results in our paper.

We also provided more randomly selected results for audio->3D and 3D->audio retrieval in the newly uploaded PDF.

Q2: Ex-MCR’s performance is lower than the compared method in some parts

For Audio-Image-Text experiments, Ex-MCR reconstructs the alignment between CLAP's audio encoder and the CLIP image encoder and text encoder. Ex-MCR outperformed ImageBind in 4 of the 6 performance metrics of Audio-Image while having much better Audio-Text alignment and the same Image-Text alignment. Therefore, comprehensively, Ex-MCR reached state-of-the-art Audio-Image-Text performance.

For 3D-Image-Text experiments, in our latest experiments(referring to the official comment to all reviewers), we use Ex-MCR to fuse more powerful models, EVA-CLIP-18B(Image-Text), and Uni3D-G(3D-Image-Text). Ex-MCR achieves leading performance in all three inter-modality alignments (3D-Image, 3D-Text, Image-Text).

Additionally, as requested by the reviewers, we provide further clarification on the 3D performance of the Ex-MCR-base. Firstly, the Ex-MCR-base demonstrates stronger Image-Text alignment and achieves a comparable level of 3D-Text alignment to ULIPv2. Therefore, overall, Ex-MCR and ULIPv2 exhibit similar alignment capabilities in 3D-Image-Text space.

We think the main reason for the performance loss in 3D-Image is the limited training data. The pseudo data for training are mainly derived from the inherent alignment of leaf space and base space. Therefore, this extending process can be regarded as distilling leaf space’s alignment knowledge to construct a new alignment for base space. The feather and ability of the distillation teacher can affect the distillation result, and when we use a stronger pre-trained space, Ex-MCR outperforms the teacher.