CSA: Data-efficient Mapping of Unimodal Features to Multimodal Features

We express our gratitude to the reviewer for their valuable feedback. We now address each point in detail.

Q1+W1: In Appendix E, we show the performance of CLIP image encoder + GTR, which partially addresses the architecture issue. In this setting, CSA still outperforms CLIP. Also, the additional experiment on text-LiDAR retrieval. CSA uses embeddings from the LiDAR encoder (Lip-loc) and GTR. It shows performance on par with LiDAR-LiDAR retrieval. In this setting, CSA utilizes the same backbone (the same encoder actually) of the LiDAR encoder.

W2+W3: We agree that ASIF is the only fair comparison, and CLIP is unfair. However, fine-tuning CLIP is not entirely fair as well, since it requires GPU and CSA does not. We additionally conducted a fair additional experiment on text-to-LiDAR retrieval (see section A in the global comment), which is the first in the community. Both the LiDAR encoder we used and CSA (text-to-LiDAR) are trained on the same KITTI train set (W3). We show that CSA achieves the same performance as the LiDAR-to-LiDAR retrieval (W2).

Per the suggestion, we used linear support vector machines (SVMs) to classify CLIP’s and CSA’s image embeddings. Note that linear classification on image embeddings is fundamentally different from cross-modal classification with text embeddings. This experiment has nothing to do with cross-modal feature space but provides us insight into the clustering of the unimodal embeddings of CLIP and CSA. The results are shown in section D in the global comment. The results indicate that the SVM trained on CLIP exhibits a 5% higher classification performance compared to CSA when using 35k training samples.

Q2: The underlying implementation of CSA is NumPy, which is efficient in handling large matrix operations (in our case SVD) parallelly on multiple CPUs. As stated in line 186 in the original draft, the time complexity of the SVD used in CSA is $O(q^1 q^2 r)$, which is roughly the cube of the unimodal encoder output dimension, not the dataset size. For most unimodal encoders, the output dimension is typically around $1000$. Also, there are tricks to speed up SVD approximately. For instance, CuPy can dramatically speed up SVD for large matrices using GPUs, and ​​​​randomized SVD algorithms compute approximate results much faster than full SVD. Also, for a given $s$, we only need to obtain the first $s$ singular values, which can also speed up the computation. Therefore, SVD is not computationally expensive in CSA.

To further support this, we provide empirical results with the machine described in the appendix. For the largest unimodal encoder size tested (DINOv2 with a dimension of $1536$), the SVD runtime was $0.55$ seconds using CuPy and $0.44$ seconds using NumPy. To sum up, we thank the reviewers for pointing this out. We will discuss more about the means to speed up SVD for large matrices in the limitations sections in the next version.

Q3: CSA does not have zero-shot capability as it requires training and testing on similar data distributions. Namely, it is not a foundation model, and it bridges multimodal data with unimodal encoders for specific tasks.

Q4: Leafy Spurge tests the limits of CSA with extremely limited data. While it is possible that CLIP or DINOv2 were trained on ImageNet, the exact training data for them are not fully revealed. To address this, a key aspect of Leafy Spurge is that it is released after the models we used, allowing for a fair comparison between CSA and CLIP on a dataset on which no unimodal encoder has been specifically trained. Specifically, we aim to isolate the impact of the task from the influence of unimodal encoders and CSA.

Minor observations:

1. In Table I, we incorrectly attributed CLIP to 12B “samples seen” but not the training size, which should be 2B. Thanks for the correction. We will update the numbers in the next version, but still, CSA requires significantly less multimodal data.

2. In Table I, we show the CLIP parameters of only one encoder for a fair comparison to other unimodal encoders (half of 2.5B is 1.2B). Thanks for the comment. We will emphasize it in the next version.

I thank the authors for addressing Q2, Q3, and Q4.

To clarify my earlier question Q1: I am suggesting that you consider evaluating your proposed method using both image and text encoders from a CLIP model that are not already aligned (e.g., taking two CLIP models pretrained on different datasets). The aim of this experiment is distinct from what you have already included in your experiments. Comparing the performance with the two original CLIP models, in my view, could further strengthen your work. Please correct me if I am wrong.

I still believe the comparison in Figure 3(a)/3(b) and Table 2(a)/2(b) is unfair and could be misleading.

Unfair comparison of Text-image tasks

As we emphasized in line 363 “ASIF is the only fair baseline for CSA… We also compared CSA with the state-of-the-art multimodal encoder model from OpenCLIP, acknowledging their incomparable training set and model parameters…” We have no intention to mislead the readers that CLIP is a fair baseline but emphasize that CLIP serves as a baseline to highlight the strong performance of CSA due to ASIF’s significantly lower performance. Also, the use of CLIP on ImageNet and Flickr30k is common in the community, which is also run in the original paper of CLIP and LAION-5B.

Lastly, in the main paper, we compare the size of multimodal training sets between CSA and CLIP to emphasize that CSA can bridge unexplored modality pairs that lack existing multimodal encoders while being performant. This motivation leads to the additional experiments detailed below.

Fair Comparison on Training Set, Unfair on Modality: Text-to-LiDAR Retrieval

Unlike the text-to-image domain, our additional experiments on text-to-LiDAR retrieval (global comment Appendix A) are completely fair in terms of train sets. CSA is on par with multimodal encoders trained on the same dataset. However, the modalities of CSA and Lip-Lock are different, thus unfair in modalities.

Beyond fairness–Remark on CSA’s contribution

CSA is a pioneering work that aims to bridge the gap between new modality pairs. Since the problem itself is new, only ASIF is a fair comparison. However, we ambitiously show that CSA can actually achieve the same performance as existing multimodal encoders despite the unfairness in train sets, modalities, and model architectures.

As suggested by the reviewer, the most we can do is conduct fair experiments per factor: (1) Appendix A in the global comment ablates the effect of train set. (2) Figure (3) and Table (2) in the main paper ablate the effect of modality pairs. (3) Appendix E in the global comment ablates the effect of model architectures.

We will clarify more in the next version of the paper and hope our explanation addresses the concerns and questions. We kindly ask the reviewer to reassess their evaluation.

We thank the reviewer for their insightful feedback.

1. Equation (9) is directly obtained from ​​the ​​previous work cited in line 242. Contrastive learning essentially enforces two properties: (1) clustering similar data instances and (2) pushing dissimilar instances far away in the feature space. Since the unimodal encoders have both properties, CSA is just re-doing “clustering similar data instances” for multimodal data pairs. CSA does that by mapping the multimodal features to maximize the coefficient correlations, which is partially the objective of contrastive learning. Hence, Appendix E in the main paper shows that CSA works well with various unimodal encoders trained on contrastive learning losses.

2. We do not consider larger-scale training datasets due to the motivation of CSA–to bridge new and unexplored modality pairs (see our additional experiment on text-LiDAR in the global comment). These modality pairs do not have sufficient data to train a cross-modal encoder. We ran the ImageNet experiment to demonstrate the capability of CSA on the image-text modality pair, and CLIP serves as an unfair baseline to understand more about CSA. In terms of the implementation of CSA on large datasets, the underlying implementation of CSA is NumPy, which is efficient in handling large matrix operations (in our case SVD) parallelly on multiple CPUs.

We thank the reviewer for their insightful feedback.

W1: As mentioned in line 473 in the main paper, we conducted an ablation study on various unimodal encoders in Appendix E. Also, our additional experiment (see section A in the global comment) on text-to-LiDAR retrieval uses a LiDAR-image encoder, Liploc, and GTR for text. It also shows the generalization ability for various unimodal encoders. As discussed in the main paper, contrastive learning-based encoders should all work smoothly with CSA, and our experimental results support this point.

W2: We agree that ASIF is the only fair comparison. We highlight that fine-tuning CLIP is not entirely fair as well, since it requires GPU and CSA does not. Also, we do not think CSA has any zero-shot capability as it requires training and testing on similar data distributions. Hence, it will likely not perform well when fine-tuned on ImageNet and then tested on Flickr. However, we conducted a fair additional experiment on text-to-LiDAR retrieval, which is the first in the community (see section A in the global comment). Both the LiDAR encoder used and CSA (text-to-LiDAR) are trained on the same train set of KITTI. This setup emphasizes the point of training the cross-modal encoder and CSA on the same dataset. We show that CSA achieves the same performance as the LiDAR-LiDAR retrieval.

W3: $q$ is not a hyper-parameter. It represents the fixed output dimensions on the unimodal encoders. $r$ is the output dimension of CCA, which does not affect the whole CSA pipeline. The only hyperparameter is $s$, which we conducted the ablation study in Appendix D. $r$ does not affect the overall results because only $s$ dimensions from the $r$ dimensional outputs are used to calculate the similarity in Eq. 4.

Q1: In Fig. 2 and section C in the global comment, we showed the t-SNE visualization of ImageNet embeddings with CLIP and CSA. We selected $20$ classes out of $1000$ classes. Both CLIP and CSA have strong clustering of embeddings, thus verifying that they have similar performance on classification.

Q2: Yes, as indicated in line 710 in the main paper, solving Equation 2 with $35,000$ multimodal feature vectors on a 64-core Xeon Gold 6226R CPU machine takes less than 10 minutes, which is significantly faster than any GPU training. Per the request of other reviewers, we also numerically showed how long it takes to solve CSA on ImageNet here: https://openreview.net/forum?id=6Mg7pjG7Sw&noteId=Siu0an4HUF

We thank the reviewers for their detailed comments.

Weakness:

1. CSA bridges multimodal data with unimodal encoders. We do not think CSA has any zero-shot capability like foundation cross-modal models. Just like most machine-learning models, it trains (solving CCA) and tests on similar data distributions. Hence, the way to use CSA is to train it with limited data of new modality pairs.

2. We highlight that the only fair comparison is ASIF, as CLIP can never be entirely fair, even with fine-tuning, which requires GPU. However, we conducted a fair experiment on text-to-LiDAR retrieval, which is the first in the community (section A in the global comment). The LiDAR encoder used and CSA (text-to-LiDAR) are trained on the same subset of KITTI. We show that CSA achieves the same performance as the LiDAR-image encoder on the retrieval task.

3. That is correct, but our additional result shows that CSA matches the performance of cross-modal encoders in text-to-LiDAR retrieval (see section A in the global comment). We will address it in the next version of the paper to clarify.

Questions:

1. No, the 5k images are also randomly sampled from the whole dataset, which means that the percentage of mislabeled images (10.9%) is the same as in Fig. 4(b). Since empirically, one does not know the mislabeled data a priori, thus we think it is more reasonable to show that the trainsets contain a fixed percentage of mislabeled data. We will address them in the next version of the paper.

2. We did not fine-tune CLIP for two reasons: (1) The essence of CSA is to bridge multimodal data with unimodal encoders without GPU training, so we do not think it is fair to fine-tune CLIP as it requires GPU. (2) We envision that CSA can be used to bridge unexplored modalities, where there are no cross-modal encoders like CLIP. The comparison to CLIP is to demonstrate that CSA is performing reasonably well. The additional text-to-LiDAR retrieval experiment emphasizes this point as there are no text-to-LiDAR encoders designed for retrieval now.

We thank the reviewer for appreciating our paper. To address the weakness:

W1: We added an additional comparison on hyperparameter $s$ vs. the AUC of detection tasks, as discussed in Section B of the global comment. The results align with the statement in the original paper: larger hyperparameter $s$ that matches the dimension of unimodal encoders is better for detection tasks. Whereas in the cross-modal retrieval task on Flickr30k, $s$ should be in the middle between the lowest and highest dimensions ($0 < 200 < 768$=GTR’s dimension). We also observe a similar trend in the additional experiment on text-to-LiDAR retrieval with the KITTI dataset (section A in the global comment). Again, we select a medium $s=100$, where the maximum dimension of the LiDAR encoder is $256$.

W2: Yes, we agree with the reviewer. We additionally tested CSA’s capability on new, unexplored modality pairs. We conducted text-to-LiDAR retrieval on an autonomous driving place recognition dataset (section A in the global comment). We showed that CSA achieves the same performance as LiDAR-to-LiDAR retrieval, showcasing CSA’s capability. Notably, we are the first to perform text-to-LiDAR retrieval, only made possible by CSA, which maps LiDAR and text embeddings into a shared feature space.

W3: Due to time constraints, we cannot run the OpenImages dataset with CSA. However, we emphasize that the underlying implementation of CSA is NumPy, which is efficient in handling large matrix operations (in our case SVD) parallelly on multiple CPUs. Using NumPy to run SVD is far more energy, time, and computationally efficient than training deep learning models on GPUs for datasets of the same size. Last but not least, there are tricks to speed up SVD. For instance, CuPy can dramatically speed up SVD for large matrices using GPUs, and ​​​​randomized SVD algorithms compute approximate results much faster than full SVD. Also, for a given s, we only need to obtain the first s singular values, which can also speed up the computation. Lastly, CSA aims to bridge unexplored modality pairs, which are naturally insufficient in data. In such cases, one does not need to worry about the computation time and the scalability of CSA. We thank the reviewers for pointing this out. We will discuss more about the means to speed up SVD for large matrices in the limitations sections in the next version.

We thank the reviewer for highlighting concerns about the time complexity of CSA. To address W3, we would like to clarify a potential misunderstanding.

Emerging Modality Pairs vs. Scalability

One important usage of CSA is to bridge new modality gaps, such as LiDAR and text. In these situations, the lack of multimodal data does not pose scalability challenges.

Time Complexity

Even when multimodal data is abundant, CSA can still operate efficiently. As stated in line 186 in the original draft, the time complexity of the SVD used in CSA is $O(q^1 q^2 r)$, roughly the cube of the unimodal encoder output dimension, not the dataset size. For most unimodal encoders, the output dimension is typically around $1000$. Therefore, SVD is not computationally expensive in CSA.

To further support this, we provide empirical results with the machine described in the appendix. For the largest unimodal encoder size tested (DINOv2 with a dimension of $1536$), the SVD runtime was $0.51$ seconds using CuPy and $0.44$ seconds using NumPy.

Hence, the scalability of CSA (or essentially SVD) should not be a major concern.

To address W2, we acknowledge that CSA is just a first step towards bridging new modality pairs, and we expect more work in the future to tackle it. However, as demonstrated in our additional experiments, we are the first to perform text-to-LiDAR retrieval, a task that previous works cannot achieve.

Regarding the number of modalities supported by CSA, it handles any modality as long as there are unimodal encoders trained with a contrastive loss for that modality. In such cases, CSA should be able to connect it with other modalities effectively.

We kindly ask the reviewer to reassess their evaluation if we have addressed the weaknesses mentioned.

Setting: We now demonstrate the effect of CSA on more modality pairs, as suggested by the reviewer. We conducted a classification of handwritten alphabets. One modality is the $3$-dimensional time series of movement of pens on a pad [2]. The other modality is either the images of alphabets or the text of "Alphabet X." We leveraged tsfresh [1] to extract statistical features from time series and re-used the same image and text encoder as the main experiments in the paper for the other two modalities. Note that tsfresh is not a contrastive-learning encoder but a statistical feature extractor, hence we also demonstrated CSA's ability to adapt any general form of encoders (feature extractors).

Results: The classification task is similar to the one of Imagenet. We calculated the AUC of multiple classification tasks (an alphabet each) and showed the average AUC under the "ovr (one-vs-rest)" setting. From Table 2 in the global comment, we see that CSA constantly outperforms ASIF by $4$~$8$% in AUC. Notably, we are the first in the community to conduct multimodal classification with multivariate time series, so there are no comparable baselines.

Remarks on modalities: We have showcased CSA's versatility across various modalities, including (1) images, (2) text, (3) audio, (4) time-series data representing 3-dimensional hand movements, and (5) LiDAR. Furthermore, CSA can be applied to other general time-series modalities, such as audio, IMU data, human body motion, and even object outlines in images [3].

We believe this comprehensive demonstration addresses the reviewer’s concerns regarding the range of modalities supported by CSA. We kindly request the reviewer to reconsider their evaluation since we have addressed all the concerns.

Reference:

[1] Christ, Maximilian, et al. "Time series feature extraction on basis of scalable hypothesis tests (tsfresh–a python package)." Neurocomputing 307 (2018): 72-77.

[2] Shokoohi-Yekta, Mohammad, et al. "Generalizing DTW to the multi-dimensional case requires an adaptive approach." Data mining and knowledge discovery 31 (2017): 1-31.

[3] Middlehurst, M. and Schäfer, P. and Bagnall, A. (2024). Bake off redux: a review and experimental evaluation of recent time series classification algorithms. Data Mining and Knowledge Discovery, online first, open access.