Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data

We thank Reviewer XDw2 for reviewing our paper and providing helpful feedback on our work. We address Reviewer XDw2’s concerns below.

Histogram of statistics revealing the space geometry

In section 3.3, it provides some statistic values to show the space geometry. I think such statistics may not actually reveal the space geometry since it averages all possible values. Is that better to provide a histogram of such statistics to demonstrate the space geometry? Furthermore, based on the constant modality gap analysis, is that the $E_{i,j}[\cos(d_i,d_j)]$ should have a value close to 1 since they should be parallel except for the noise effect.

Thank you for the suggestion and question. We have now included the histogram, as well as a detailed explanation of how to interpret these statistics in Appendix C.

For $E_{i,j}[\cos(d_i,d_j)]$, because there is a noise entangled in $d_i = c_\perp + \epsilon_i$ and $d_j = c_\perp + \epsilon_j$, the cosine similarity will not be 1. However, if we compute the group-level $E_{i,j}[\cos(d_{c_i},d_{c_j})]$, where $c_i$ and $c_j$ are 100 randomly sampled images, the value reaches 0.99. We provide a detailed explanation in Appendix C.

Why corrupt ($C_2^2$) seems more effective than collapse ($C_1^2$)?

The abstract and introduction emphasize the importance of the modality gap of the feature space, however, the results in the experiments seem show that the key component is the alignment noise. For instance, in Table 2, $C_2^2$ (only use connect and corrupt) can achieve a similar performance as the $C^3$. Compared to $C^1$, only adding collapse boost 16.3 but 41 when only adding alignment noise.

Thanks for the question. While $C_2^2$ (corrupt) seems to be more effective than $C_1^2$ (collapse) in Table 2 (image captioning on CLIP embedding space), they seem to be similarly effective in Table 3 (text-to-image generation on CLIP embedding space) and Table 4 (image / video / audio captioning on ImageBind embedding space).

We hypothesize that the greater effectiveness of $C_2^2$ is because $C_2^2$ injects random noise in the embedding space during training, including the direction of the modality gap, which potentially reduces the model's sensitivity to this gap. Nevertheless, given the substantial size of the modality gap, $C_1^2$ remains necessary to diminish this gap, thereby enhancing the overall performance of $C^3$.

It is also worth noting that $C^3$ adds near-zero additional computational cost over baselines. As our experiments demonstrate its effectiveness across a wide range of tasks, it can be a template for future work.

Significance of the quantitative improvement and more qualitative examples

The experiments are conducted on generation tasks, the quantitative performance is similar for the method and baseline, while in the qualitative examples, I can not tell which method is better based on three examples. It will be better to provide more qualitative examples.

Thank you for the suggestion. We have added more qualitative examples for image captioning and text-to-image generation in Appendix Figure 10 and Figure 11. We hope these examples provide additional evidence of our method’s effectiveness.

We also emphasize that the improvement of our method is significant. Per Reviewer bNKP’s suggestion, we have run $C^3$ in Table 2 three times with random seeds 1234, 5678, and 910 and reported the numbers below. There is a very small variance across different runs, and $C^3$ consistently outperforms all the baselines.

Dear reviewer XDw2,

Thank you again for providing us with constructive feedback. We appreciate your consideration of our response and willingness to adjust your rating based on our improvements!

Here we provide additional clarification to address your concerns:

For Q1, I realize why the modality gap is constant if the image and text share no effective dimensions. And the example just follows that there are no overlapped effective dimensions. But are that some propositions to guarantee this phenomenon?

Thanks for the question. We hope to clarify that the theory still works when image and text share effective dimensions.

Here we provide a modified example to help understand the process:

Suppose there are two image-text pairs in 3D space; both the 1st and 2nd dimensions of the image embeddings are effective, and the same applies to text embeddings, and their 3rd dimension both are ineffective.

During initialization, let us denote these embeddings as $x_1, x_2, y_1, y_2 \in \mathbb{R}^3$:

$x_1 = [0.7, 0.5, -0.5]$ (the 3rd dimensions are constant across $x_i$)

$x_2 = [-0.7, -0.5, -0.5]$ (the 3rd dimensions are constant across $x_i$)

$y_1 = [0.5, 0.7, 0.5]$ (the 3rd dimensions are constant across $y_i$)

$y_2 = [-0.5, -0.7, 0.5]$ (the 3rd dimensions are constant across $y_i$)

During optimization, only the union of effective dimensions (i.e., 1st and 2nd) will be aligned, and the intersection of ineffective dimensions (i.e., 3rd) will remain constant:

$x_1 = [0.6, 0.6, -0.5]$ (the 3rd dimension remains constant because of no gradient)

$x_2 = [-0.6, -0.6, -0.5]$ (the 3rd dimension remains constant because of no gradient)

$y_1 = [0.6, 0.6, 0.5]$ (the 3rd dimension remains constant because of no gradient)

$y_2 = [-0.6, -0.6, 0.5]$ (the 3rd dimension remains constant because of no gradient)

Therefore, we have a constant gap with a distance of 1.0, which is also orthogonal to the embedding spans.

Instead, the key assumption of the theory is that image and text share ineffective dimensions, which is empirically verified in various models such as CLIP. For example, in CLIP, the image has only 25 effective dimensions, and the text has only 230 effective dimensions. Therefore, the image and text must share ineffective dimensions in 512D space, as they sum up to at most 255 effective dimensions. Therefore, these ineffective dimensions will be fully determined by initialization, and will not change during optimization, creating a constant orthogonal modality gap.

Please let us know if you have further questions or concerns!

We thank Reviewer CpsN for their positive comments and for providing thoughtful feedback on our work. We address Reviewer CpsN’s concerns below.

Novelty of $C^3$ algorithm

The proposed $C^3$ algorithm has limited novelty on its own given that each step has been studied in previous work. However, the combination of these steps is new and well-motivated by the theoretical framework developed in this paper, which mitigates the lack-of-novelty issue.

Thank you for the question. Yes, as you mentioned, while existing works have empirically proposed collapse and corrupt operation separately, we differ from them in three aspects:

1. Our method is built on the theoretical understanding of multi-modal contrastive representation space geometry.
2. We unified these insights and showed the combination of them leads to superior performance than each of them.
3. We proved the effectiveness of our method in a wide range of tasks, modalities, data, and embedding spaces, which can become a standard recipe for all future works built on multi-modal contrastive embedding space.

Collapse operation implementation

How is the "Collapse" step implemented (i.e., computing e_x' and e_y')? Is it the same as batchnorm?

Thanks for the question. The “collapse” step can be viewed as a distribution norm, where the image embedding mean $\bar{e}_x$ and text embedding mean $\bar{e}_y$ are pre-computed on the entire training set and subtracted during training and inference. The batch norm is an approximation with a larger variance. We provided an algorithm formulation of $C^3$ in Appendix C for further clarification.

Codebase

The current submission does not include code, hopefully the authors can release them later to facilitate future research.

Thanks for the question. We included an anonymous GitHub link in the Page 10 reproducibility statement, which should be able to reproduce all the results. Reviewer Vv1o has checked this codebase and commented that “the codebase looks carefully developed and seems free from glaring bugs”.

Thank you again for your feedback, which is very helpful in improving the paper. We hope the above responses and changes to our manuscript adequately address your concerns. Please let us know if you have further questions or concerns!

We thank Reviewer Vv1o for their positive comments and for providing thoughtful feedback on our work. We address Reviewer Vv1o’s concerns below.

More discussion about Table 1

The discussion on the alignment noise could be improved: in particular, the results in Table 1 are left for the reader to infer. The same statistics could also be easily computed upon any other modality combination in the appendix, it would be useful to see if it still applies.

Thank you for your suggestion. We have now included a detailed explanation of how to interpret these statistics in Appendix C. We also included statistics of embeddings from different modalities in Appendix C. We hope these edits will help readers easily understand Table 1.

Would reduce the dimensionality mitigate the modality gap?

Since the modality gap is due to the dimensional collapse, would reducing the dimensionality to the effective one help overcoming the issue?

Thanks for the question. To understand how dimensionality affects the gap, we initialized CLIP with different dimensions. We find that, in lower dimensions, the ratio of effective dimensions increases for both encoders (but cannot reach 100%) and the modality gap decreases.

Nonetheless, we hope to clarify that while the relationship between the dimensionality and the gap is an interesting question, this gap can be well addressed by the collapse operation based on our theoretical analysis. Therefore, the gap actually does not matter.

Connection to [1]

Is there any relation between the decomposition of the modality gap with the content-style-modality specific decomposition assumed e.g. in [1]? Briefly, each latent vector in a multi-modal contrastive learning space is there assumed to have a part that is shared across modalities, i.e. the content, one that is shared but with some distortion, i.e. the style, and one that is not shared at all, i.e. the modality-specific component. Is it possible that the modality specific component in [1] is just the constant component caused by the different initializations seen in this work?

Thank you for pointing out this interesting work. The modality-specific component in [1] appears to be more closely related to the "alignment noise" than the "modality gap". For instance, [1] identifies "object rotation" as a modality-specific component in the visual domain. This suggests that changes like rotating an image would solely affect image embeddings, with no parallel effect in text embeddings. Therefore, this phenomenon seems to align more with alignment noise, where image and text embeddings are not perfectly aligned in the shared representation space.

Can modality gap differ in scale?

The solution to the modality gap is to center the embeddings, implying the modality gap is just a shift. Isn’t it possible that the difference in modality may also result in different scales?

Thanks for your question. Based on our theoretical analysis, the modality gap can only be a shift, not a scale difference. Our analysis reveals that the embeddings’ effective dimensions from different modalities will be aligned, and the ineffective dimensions will remain constant, which creates a constant gap. If there is a scale difference, gradients will be propagated into the effective dimensions and make the scale the same.

Thank you again for your feedback, which is very helpful in improving the paper. We hope the above responses and changes to our manuscript adequately address your concerns. Please let us know if you have further questions or concerns!

References

[1] Daunhawer, I., Bizeul, A., Palumbo, E., Marx, A., & Vogt, J. E. (2022, September). Identifiability Results for Multimodal Contrastive Learning. In The Eleventh International Conference on Learning Representations.

We thank Reviewer bNKP for their positive comments and for providing thoughtful feedback on our work. We address Reviewer bNKP’s concerns below.

Novelty compared to existing works

My major concern is about novelty. [1] has pointed out that random initialization and contrastive learning causes and preserves modality gap, and [2] has modeled modality gap as a constant orthogonal to image and text span. The proposed C^3 is also an ensemble of existing methods [2][3][4]. Especially, the cross-modal transferability in [2] seems quite similar to the ``interchangeability of embeddings from different modalities’’ in this paper. Please justify.

Thank you for the question. While some previous works have found the modality gap exists and empirically observed that addressing the gap can improve performance, no paper so far can provide a comprehensive and complete theoretical understanding of why there is a gap, how the geometry looks like, and what we should do for the gap. In this work, we provide a unified theoretical understanding of the gap and provide a simple yet effective method that addresses the gap in a principled way. Our experiments demonstrate its effectiveness across a wide range of tasks, which can be a template for future work.

Here we further provide a detailed comparison with previous works, which is also included in Appendix A:

In terms of theory about multi-modal representation space geometry:

[1] explained there is a modality gap caused by initialization and optimization, but its theory failed to justify its geometry.

[2] empirically identified the modality gap geometry, but it failed to explain how the geometry arises.

We are the first work that explains how the geometry arises. We identified the dimensional collapse phenomenon in initialization and the collapsed gradients in optimization, both of which are not explored by [1,2], and their combination well explained that the modality gap geometry would be a constant orthogonal vector. We also connect dimensional collapse back to the cone effect proposed in [1] (Appendix F.2), while the former is a more general and well-explored analytical method.

In terms of methods for enhancing the interchangeability of embeddings from different modalities:

[2] proposed the collapse operation (i.e., removing embedding mean from each modality) in classification settings, based on an empirical understanding of the modality gap geometry.

[3,4] empirically found that the corrupt operation (i.e., adding noise to embeddings) works in generation settings.

Our work unified these insights and showed the combination of them leads to superior performance than each of them. Our method is built on our solid theoretical understanding of multi-modal representation space geometry. We proved our method's effectiveness in a wide range of tasks, modalities, data, and embedding spaces, which can become a standard recipe for all future works built on multi-modal contrastive embedding space.

In terms of the relation between cross-modal transferability and embedding interchangeability:

Embedding interchangeability is a more fundamental concept that enables cross-modal transferability on both classifiers (demonstrated in [2]) and generators (shown in this work). Therefore, in this work we focus on enhancing embedding interchangeability.