We appreciate your positive comments and insightful suggestions.

Details of text-only CL in Section 2.1 The key process is similar to that described in Section 3.1. We first decouple the LLM from the MLLM and train it using a standard contrastive loss (InfoNCE loss), where each positive pair consists of two texts; we then plug the LLM back to the MLLM after training. For the experiments in Section 2.1, we use the all-NLI dataset, in which each positive pair is composed of an anchor and an entailment from a natural language inference dataset; for example, Anchor: "Children smiling and waving at the camera"; Entailment: "There are children present." All other texts within the same batch are treated as in-batch negatives. In contrast, for the main experiment in Section 3.1, we employ an additional dataset we constructed, Scale1M, which incorporates a diverse range of multilingual data sources. We appreciate your suggestion and will enrich the descriptions in Section 2.1 accordingly!

Performance of Full-Finetuning vs. LoRA We provide full-finetuning results as follows, which provides decent performance but lags behind LoRA training, when trained with the same Qwen2.5-VL 3b backbone.

We hypothesize that full-finetuning introduces more perturbation to the pretrained parameter space, which stores latent cross-modal alignment. In contrast, LoRA beter preserves this alignment through minimal adaptation of model parameters.

The experiment in Section 4.2 The experimental results discussed in Section 4.2 are visualized in Figure 5. After establishing the theoretical relationship between the generative and representational capabilities of MLLMs, we compare the base models' performance on generative benchmarks with their post-contrastive learning performance on embedding benchmarks, showing a positive scaling correlation between generation and representation capability, supporting our theoretical analysis.

Missing citations Thanks for spotting the typo. We will add the citation to LLaMA in Line 97 in the revised version.

Thank you for your valuable feedback. We address your comments point by point as follows:

• Comparison with native Qwen-VL/Omni embeddings, and well-known cross-modal alignment methods:

We present a comparison of results between the native Qwen2.5-VL embeddings, a CLIP-style training baseline, and our text-only CL method on vision-and-language embedding tasks in the Table below. For the CLIP-style baseline, we fine-tune the native Qwen2.5-VL-3b model using 800k image-caption pairs from the PixmoCaps dataset with contrastive learning. The results of our text-only contrastive learning method, which is trained on the all-NLI dataset (270k pairs), are also reported. Experiment results show that our text-only method consistently improves the performance of the native Qwen embeddings across all tasks. Moreover, our method outperforms the CLIP-style baseline, while requiring only about 1/50th of the training time.

The following table also presents the comparison results of Qwen2.5-Omni embedding model before and after finetuned by our text-only CL method on vision-and-language embedding tasks. We will include the results of the CLIP-style baseline in the final version.

The following tables present the performance of Qwen-VL and Qwen-Omni before and after our text-only CL method on video retrieval and audio retrieval tasks. For the results of video retrieval, we report the average of the t2v and v2t recall@5 scores on the msrvtt and ActivityNet datasets. For the results of audio retrieval, we report the average of the t2a and a2t recall@5 scores on the Clotho and AudioCaps datasets.

• Visualizing collapsed vs isotropic space

We have created visualizations using t-SNE and UMAP data projections. Unfortunately, PDF uploads and link sharing are not permitted in the rebuttal. We will include these visualizations in the camera-ready version given the chance.

We would also like to highlight that we have provided quantitative results on the degree of anisotropy in Section 2.1. Measuring the expected cosine similarity between random samples is a validated method for approximating anisotropy degree, with lower values indicating higher isotropy [1]. Our results demonstrate that our text-only contrastive learning method effectively mitigates anisotropy, thereby enhancing the isotropy of the embedding space.

• LoRA training vs other methods (e.g., Projection Layer or Full-Finetuning)

We compare LoRA with a full-finetuning method and a shallow projection baseline that only adds a linear projection layer after the output layer. The results of Qwen2.5-VL-3b are presented in the following table. The shallow projection baseline introduces additional parameters to the model, but does not fully leverage the pretrained latent cross-modal alignment, resulting in only a modest average improvement over native embeddings. While full fine-tuning yields reasonable performance, it still lags significantly behind LoRA training. We posit that this is because the contrastive loss used during fine-tuning differs substantially from the pretraining loss (next word prediction) of MLLM, causing full fine-tuning to introduce large perturbations to the model’s pretrained parameters. These perturbations are more likely to disrupt the model’s pre-established cross-modal alignment.

Furthermore, full-finetuning is much more compute-intensive for contrastive-learning given the quadratic GPU RAM requirements of contrastive learning loss over batch size. e.g., it would take 16 GPUs of 96G RAM to perform the same full-finetuning experiment on 7B models.

[1] Ethayarajh 2019, How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings

Thanks for your detailed comments. We aim to address your concerns as follows:

1. Clarifying Core Claim

We clarify the logic between our experiment findings and the core claim as follows:

Our experiments demonstrate that applying text-only contrastive learning to MLLMs not only improves the isotropy of the embedding space in the text modality, but also generalizes to enhance isotropy in non-textual modalities—even without finetuning on multi-modal data. The successful transfer of these improvements across modalities provides empirical evidence that MLLMs inherently maintain geometrically aligned latent spaces among different modalities. We will emphasize this finding in the revised version.

We have provided quantitative results on the degree of anisotropy in Section 2.1. The degree of anisotropy is a standard metric for approximating isotropy, with lower values indicating higher isotropy. In lines 72 to 75, we mentioned that "we examine whether this text-specific fine-tuning effectively mitigates anisotropy in non-textual modalities without requiring explicit multimodal training. Successful transfer of improvements across modalities would provide empirical evidence that MLLMs inherently maintain geometrically aligned latent spaces across different modalities."

2. Whether other modality features improve

We provide extensive results on the performance of our method in embedding tasks across different modalities in Figure 4, 5 and Table 1, including both uni-modal tasks (image-only: linear probing and visual STS) and cross-modal tasks (image-text: document understanding, multilingual image retrieval; audio-text: audio retrieval; video-text: video retrieval). In the table below, we present the embedding performance of the MLLM before and after text-only contrastive finetuning to enable a direct comparison.

We would like to highlight that, cross-modal retrieval performance would have degraded if non-textual modality spaces do not follow the enhanced textual representation space geometrically.

3. Questions for Section 2.1 Anisotropy Analysis

First, we want to highlight that the lower is the better for anisotropy estimate value, as we are measuring the expected cosine similarity of random examples, instead of paired examples.

‒ Setting. We use 1000 examples for each modality (1000 texts & images, 1000 texts & audios, 1000 texts & videos ) for the 3 settings, respectively, leading to 499,500 cosine similarity values (lower triangular of the similarity matrix) for each setting, and we take the mean for it. This is typically deemed sufficient for approximating anisotropy [1].

‒ Why cosine similarity decreases every layer. Note that as we are measuring cosine similarity expectation of random pairs, and it is a good property for random pairs to have lower similarities (more isotropic, less anisotropic). As the layer progresses, the model contextualizes each input more, leading to more different embeddings for each input, and thus, their cosine similarity with other inputs would become lower.

‒ "Qwen-omni is anisotropic but why good on benchmarks?" After text-only contrastive learning, the expected cosine similarity goes down (less anisotropic) for all modalities, making the embedding space more isotropic. We show in the paper that this leads to state-of-the-art performance on embedding tasks.

4. Questions for Section 2.2 Kernel Analysis

‒ Why Qwen-VL 3b has more layers than Qwen-VL 7b: According to Table 1 of the Qwen2.5-VL technical report [2], Qwen-VL 3B has more layers in the decoder LLM than Qwen-VL 7B (36 layers vs. 28 layers). The report does not provide an explicit rationale for this architectural decision. We suspect that this configuration may lead to better overall performance.

‒ Improvement is small This section measures cross-modal alignment, instead of the actual benchmark performance. We intend to show that, through text-only contrastive learning, cross-modal alignment actually improves, as opposed to decreasing.

5. Performance gain over base models.

In point 2 above, we provided extensive results comparing base models.

6. Missing citations

Thanks for spotting the typo. We will add the citation to LLaMA in Line 97 in the revised version.

[1] Ethayarajh 2019, How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings

[2] Qwen2.5-VL Technical Report. Qwen Team et al. 2025.

Thank you for your constructive comments. We address your high-level concerns as follows:

Open SOTA and API SOTA: Thanks for the suggestion. "Open SOTA" and "API SOTA" refer to E5-V and Voyage-multimodal-3, respectively, as cited in line 222 and the caption of Figure 4. To improve clarity, we will also update the legend in Figure 4 to display the actual model names.

What Figure 3 expresses: Figure 3 illustrates our proposed method. This figure is intended to outline how text-only contrastive learning enhances the quality of omni-modal representations. Specifically, text-only contrastive learning can effectively mitigate anisotropy in non-textual modalities, even in the absence of explicit multimodal training. We conduct empirical studies in Section 2.1 to demonstrate this effect. As shown in Figure 1, text-only contrastive learning reduces the expected cosine similarity of random text-image, text-audio, and text-video pairs from over 0.8 to around 0.5. This indicates that anisotropy is reduced and the embedding space becomes more discriminative.

Case studies: We find that conducting text-only contrastive learning significantly improves the model’s fine-grained cross-modal alignment, enabling it to better distinguish between hard-negative texts. For instance, the models perform much better on compositionality evaluation tasks—such as examples from the ARO benchmark—where the goal is to match the correct, fine-grained caption to each image rather than its hard negative. As an example, consider a picture of a man sitting to the right of a door, with the following two fine-grained descriptions:

• "The door is to the left of the shirt" (correct)
• "The shirt is to the left of the door" (incorrect)

Previously, the model would assign a higher cosine similarity to the incorrect caption. After applying text-only contrastive learning, however, it correctly assigns higher similarity to the accurate caption. We appreciate your suggestion and will include such case studies in the paper!