On the Comparison between Multi-modal and Single-modal Contrastive Learning

References

[1] Allen-Zhu, Zeyuan, et al. A convergence theory for deep learning via over-parameterization. In International conference on machine learning. 2019.

[2] Allen-Zhu, Zeyuan. Learning and generalization in overparameterized neural networks, going beyond two layers. In Advances in neural information processing systems. 2019.

[3] Cao, Yuan, et al. Benign overfitting in two-layer convolutional neural networks. In Advances in neural information processing systems. 2022.

[4] Meng, Xuran, et al. Benign Overfitting in Two-Layer ReLU Convolutional Neural Networks for XOR Data. In International Conference on Machine Learning. 2024.

[5] Wen, Z. and Li, Y., 2021, July. Toward understanding the feature learning process of self-supervised contrastive learning. In International Conference on Machine Learning (pp. 11112-11122). PMLR.

[6] Xu, Z., Wang, Y., Frei, S., Vardi, G., & Hu, W. (2023). Benign overfitting and grokking in relu networks for xor cluster data. ICLR 2024.

[7] Kou, Yiwen, et al. "Benign overfitting in two-layer ReLU convolutional neural networks." International Conference on Machine Learning. PMLR, 2023.

We would like to thank the reviewer for their detailed and constructive feedback on our paper. Below, we provide detailed responses to the main comments

---

W1(a) Incorrect use of big-O-like notations.

R1(a) We appreciate the reviewer's attention to the details of our notation. We have followed standard usage of big-O notation, such as in [1,2,3,4]. Big-O notation is indeed a mathematical tool used to describe the asymptotic behavior of functions. While it is most commonly used to describe functions in terms of variables, it can also be used to describe constant bounds. For instance, the notation $O(1)$ is widely accepted and used to indicate a constant upper bound regardless of the size of the input.

---

W1(b) $d>n^2$ is unrealistic.

R1(b) We appreciate the reviewer's concern regarding the assumption $d> n^2$. Our work seeks to provide a theoretical understanding of the difference between multi-modal and single-modal contrastive learning in the over-parameterization regime. The high dimensional setting we adopted ensures sufficient over-parameterization, as the number of trainable parameters is $d \times m$. This setting is common in theoretical studies to simplify analysis and highlight key phenomena. Similar high-dimensional settings have been adopted in relevant prior works [3,4,5,6].

---

W1(c) Undefined variable σ_0, n, η.

R1(c) Thank you for bringing this to our attention. $\sigma_0$ represents the initialization strength of the weights of the neural network, where $\mathbf{w}^{(0)}_r \sim \mathcal{N}(0, \sigma^2_0 I)$. $n$ is the number of training samples and $\eta$ is the learning rate. We will ensure that these variables are explicitly defined and clearly explained in the revised version of the manuscript.

---

W1(d) Unclear how assumption (6) distinguishes between the single-modal and multi-modal features.

R1(d) Assumption (6) specifically applies to the first modality's SNR. Assumption (7), on the other hand, specifies the relationship between the signal strengths of the two modalities, given that we assume the noise scale is the same across both modalities (as stated on Line 123). Together, assumptions (6) and (7) establish the requirements on the SNR for the second modality, thereby distinguishing the roles of single-modal and multi-modal features.

---

W2 ϵ is a vector according to Eq(5), but is scalar in Theorem 4.2.

R2 In our paper, we use bold-faced letters to denote vectors and matrices, while non-bold letters represent scalars (line 106). Therefore, the scalar $\epsilon$ and the vector $\boldsymbol{\epsilon}$ are distinct variables. We will ensure to clarify this notation in the revised version of the manuscript to avoid any confusion.

---

W3 Confusing definition of the loss derivatives in Eq (8).

R3 We use $\ell'^{(t)}_i$ to denote the 'effective' loss derivative, which is a component of the gradient of $L$ with respect to $L$ with respect to $\mathbf{w}_r^{(t)}$. The complete form of gradient is given in Eq. (10) in the main paper. We will ensure to clarify this notation in the revised manuscript to avoid any confusion.

---

W4 Lemma 5.1 is not proven.

R4 Lemma 5.1 is derived directly from the weight decomposition (Eq 11) and gradient descent training (Eq 7). Similar results can be found in the literature [3,7]. We will include the proof of Lemma 5.1 in the appendix of the revised manuscript to ensure completeness and clarity. Thank you for pointing this out.

---

W5: Hard to find where Lemma 5.2 is proved in the Appendix.

R5: Thank you for the comments. We will make sure to restructure the proof to improve the clarity and add explanations for each lemma. For the proof of Lemma 5.2, it can be found in Appendix C.1.3. Apart from the contents of the appendix which we have already provided, we will add references to Lemma 5.2.

---

W6: The roadmap in Section 5.1 does not explicitly tell us how we end up with Theorem 4.2.

R6: We would like to highlight that proof roadmaps are to provide major steps and lemmas (rather than all of them) that lead to a final conclusion. And we leave the remaining part in the appendix given the limited space in the main paper. In our revised version, we will add more explanations by including the remaining proof sketches that lead to Theorems in Section 4.

Here we make an explanation using single-modal learning (Section 5.1) as an example. After we have shown convergence of the dynamics in the second stage, we can show the maximum signal learning is on the order of $\tilde O(1/\sqrt{n})$, while maximum noise memorization is on the constant order (Lemma C.17). This means for downstream tasks, the embedding of a test data would be dominated by the noise, overshadowing the signal. Thus the embedding is not linearly separable, which gives a constant test loss.

---

W7: Line 560, it is unclear what "RHS" is referring to.

R7: the RHS denotes RHS of the inequality

$\mathbb{P} \big( |\langle \mathbf{w}^{(0)}_{r} , \boldsymbol{\mu} \rangle| \leq c \big) \leq 2 \sqrt{1 - \exp\big( - \frac{2 c^2}{\sigma_0^2 \\| \boldsymbol{\mu} \\|_2^2 \pi} \big)}$

The $\delta$ here is defined in line 558 of Lemma B.2. In the proof of Lemma B.2, it is the first time to use $\delta$, thus it is valid. Similarly, the $\delta$s in Lemma B.3, B.4, B.5 are all defined within the scope of the respective lemma.

---

W8:It is also unclear how the "noise memorization" and "signal learning" are defined in the experiment in Figure 1.

R8: The signal learning and noise memorization refer to the coefficients $\gamma_r^{(t)}$ and $\rho_{r,i}^{(t)}$, which we have already explicitly defined in Line 214. In the experiments, we track the coefficients along the optimization path to plot the results.

---

Q: Does your theorems provide any new insight into future research contrastive learning?

A: Please refer to the global rebuttal

Thank for your the follow-up question. Below, we provide a line-by-line proof for Lemma B.2.

Proof of Lemma B.2. By Lemma B.1 (anti-concentration result for Gaussian random variable), and because $\langle \mathbf{w}^{(0)}_r, \mathbf{\mu} \rangle \sim \mathcal{N}(0, \sigma_0^2 \| \mathbf{\mu} \|^2_2)$, we can show

$$\mathbb{P} \big( |\langle \mathbf{w}^{(0)}_{r} , \mathbf{\mu} \rangle| \leq c \big) \leq 2 \sqrt{1 - \exp\big( - \frac{2 c^2}{\sigma_0^2 \| \mathbf{\mu} \|_2^2 \pi} \big)}.$$

We first set $c = 100 \cdot \text{SNR} \sqrt{\frac{8\log(6n/\delta)}{d}} n$ and plug it into the RHS of the above inequality, which becomes

$\text{RHS} = 2 \sqrt{1 - \exp\big(- \frac{16\times100^2 \cdot \log(6n/\delta) n^2}{\sigma_0^2 \sigma_\xi^2 d^2 \pi} \big)}.$

Then we can verify that when $d$ satisfies the condition on Line 561, i.e., $d \geq \frac{400n}{\sigma_0 \sigma_\xi} \sqrt{\frac{\log(6n/\delta)}{-\pi \log(1- \delta^2/(4m^2))}}$, we can verify that

$\text{RHS} \leq \frac{\delta}{m}.$

This suggests for a single neuron $r \in [m]$, we have

$\mathbb{P}( |\langle \mathbf{w}_r^{(0)}, \mathbf{\mu} \rangle| \leq c ) \leq \frac{\delta}{m}.$

Taking the union bound for all the neurons $r \in [m]$, we have

$$\leq \sum_{r=1}^m \mathbb{P}( |\langle \mathbf{w}_r^{(0)}, \mathbf{\mu} \rangle| \leq c ) = \delta.$$

This then implies that

$\mathbb{P}( \cap_{r =1}^m \{|\langle \mathbf{w}_r^{(0)}, \mathbf{\mu} \rangle| \geq c \} ) \geq 1 -\delta.$

This concludes that with probability at least $1-\delta$, for all neurons $r \in [m]$, we have $|\langle \mathbf{w}_r^{(0)}, \mathbf{\mu}\rangle| \geq c = 100 \cdot \text{SNR} \sqrt{\frac{8\log(6n/\delta)}{d}} n$.

We hope the above line-by-line proof resolves your question and we believe the wording "set RHS to $\delta/m$" is what has caused the confusion. We will remove such wroding and make the proof clearer in the revised version.

If you have any further questions, we are more than willing to address them. If you find that our responses have sufficiently addressed your concerns and questions, please consider increasing your score. Thank you for your time.

We would like to thank the reviewer for their constructive feedback and thoughtful comments on our paper. We appreciate the opportunity to address the points raised and clarify our contributions. Below, we provide detailed responses to the main comments and questions.

---

W1: Strong assumptions made in the theoretical analysis The analysis relies on some strict assumptions, such as specific signal-to-noise ratios and network initialization conditions, which might not hold in real-world applications.

R1: The data model adopted in this work aligns with related works in the field. Specifically, we use small weight initialization from a Gaussian distribution, ensuring that the network with gradient descent can effectively perform feature learning, as demonstrated in [a,b]. If large initialization were used, the training might fall into the lazy training regime or neural tangent kernel (NTK) regime, which has different learning dynamics and generalization properties [c,d]. The specific SNR values were chosen to create a scenario where we can make a meaningful comparison between single-modal and multi-modal contrastive learning. Similar strategies have been adopted in prior works [e,f].

---

W2: Some influential previous works are not introduced such as UMT/UME[1], QMF[2], MMParato[3], and ReconBoost[4].

R2: We thank the reviewer for pointing out these valuable references. Indeed, the mentioned works provide significant theoretical and empirical insights into multimodal learning:

[1] "On Uni-Modal Feature Learning in Supervised Multi-Modal Learning" (UMT/UME) explores feature learning in multimodal contexts, although its setting and claims differ from ours.

[2] "Provable Dynamic Fusion for Low-Quality Multimodal Data" (QMF) focuses on a popular multimodal fusion framework from a generalization perspective.

[3] "MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance" identifies previously ignored gradient conflicts between multimodal and unimodal learning objectives through optimization analysis.

[4] "ReconBoost: Boosting Can Achieve Modality Reconcilement" explores a novel multimodal alternating learning paradigm that aims to reconcile the exploitation of unimodal features with the exploration of cross-modal interactions.

These studies are relevant to our work, and we will cite them and add a discussion in our revised version to provide a more comprehensive overview of the related literature and to position our contributions in the broader context of multimodal learning research.

---

Q: The theoretical analysis is predicated on rather strong assumptions. If the volume of data and the complexity of the models are significantly increased, would conclusions still be applicable?

A: Our theoretical analysis indeed can cover scenarios with large data sizes (when sample size $n$ and dimension $d$ increases). This ensures that our conclusions about training dynamics and generalization hold even with an increase in the volume of data.

We believe that the main insights regarding the benefits of modality cooperation and the importance of signal-to-noise ratio (SNR) are likely to extend to more complex data distributions (such as non-linearly separable) as well as more complex model architecture (such as transformers). The core principles of multi-modal learning, such as leveraging complementary information from different modalities to enhance signal learning, are not inherently tied to the simplicity of the data model. We acknowledge that more complex scenarios require further investigation and we consider this an interesting direction for future work.

---

References

[a] Cao, Yuan, et al. "Benign overfitting in two-layer convolutional neural networks." Advances in neural information processing systems 35 (2022): 25237-25250.

[b] Suzuki, Taiji, et al. "Feature learning via mean-field langevin dynamics: classifying sparse parities and beyond." Advances in Neural Information Processing Systems 36 (2024).

[c] Jacot, Arthur, Franck Gabriel, and Clément Hongler. "Neural tangent kernel: Convergence and generalization in neural networks." Advances in neural information processing systems 31 (2018).

[d] Chizat, Lenaic, Edouard Oyallon, and Francis Bach. "On lazy training in differentiable programming." Advances in neural information processing systems 32 (2019).

[e] Wen, Zixin, and Yuanzhi Li. "Toward understanding the feature learning process of self-supervised contrastive learning." International Conference on Machine Learning. PMLR, 2021.

[f] Zou, Difan, et al. "Understanding the generalization of adam in learning neural networks with proper regularization." arXiv preprint arXiv:2108.11371 (2021).

We would like to thank the reviewer for their constructive feedback and insightful comments on our paper. We appreciate the opportunity to address the points raised and clarify our contributions. Below, we provide detailed responses to the main comments and questions.

---

W1: Can you provide intuition for why the cooperation between modalities leads to better signal learning in the multi-modal case? How might this insight be leveraged to improve single-modal contrastive learning?

R1: Both the single-modal and multi-modal contrastive learning aim to maximize the similarity between positive pairs while minimizing the similarity between the negative ones. The difference is single-modal learning forms positive pairs by data augmentation while multi-modal learning forms positive pairs by the correspondence between the two modalities. Usually, the data augmentation does not change the signal-to-noise ratio (SNR), while the two modalities can have different SNR. Due to the use of contrastive loss, the learning of signals in one modality highly depends on the signal learning in the other modality. Thus, when the other modality has a higher SNR, the learning of the first modality is lifted by aligning with the second modality. This cooperative learning leads to better overall signal learning compared to single-modal contrastive learning.

To improve single-modal contrastive learning, one could potentially simulate the effect of multi-modal learning by augmenting the single modality with additional synthetic views that enhance the signal-to-noise ratio, thereby improving the overall learning process.

---

W2: Do your theoretical insights suggest any practical strategies for improving multi-modal contrastive learning?

R2: Please refer to the global rebuttal

---

W3: Do you expect the main insights to hold for more complex data distributions?

R3: While our current analysis is based on a linear data generation model, we believe that the main insights regarding the benefits of modality cooperation and the importance of signal-to-noise ratio (SNR) are likely to extend to more complex data distributions. The core principles of multi-modal learning, such as leveraging complementary information from different modalities to enhance signal learning, are not inherently tied to the simplicity of the data model.

To ensure broader applicability, future work will focus on validating these insights with non-linear and more realistic data models. This will involve experimenting with a variety of data distributions and network architectures to confirm that the advantages of multi-modal cooperation and improved SNR hold true in more practical and complex scenarios.

---

W4: The experimental validation is limited to synthetic data.

R4: We have extended the comparison to realistic image data, ColoredMNIST [1,2], which is a typical benchmark studying the generalization capability under distribution shifts. The task is a 10-class classification that recognizes the number of the colored MNIST images. The two modalities are image and text that describes the images. We implement the multi-modal learning following the practice of [2], where we consider an ideal language encoder that successfully encodes the caption of the images into one-hot labels of colors and digits. For single-modal learning, we follow the implementation of the SimCLR paper to construct a set of augmentations to learn the representations. Then, under a mild and realistic distribution shift, we verify that CLIP archives an out-of-distribution test accuracy of 82.13%, which outperforms that of SimCLR 12.68%.

---

Q: Line 205: “On the contrary, augmentation often maintains the same SNR as the original data, so single-modal learning hardly benefits from the augmentation and can only memorize the noise from the data.” This claim significantly contradicts empirical experiments such as SimCLR and MoCo. How would you justify this?

A: Our argument does not contradict the success of single-modal contrastive learning methods such as SimCLR and MoCo. The statement is made in a comparative context with multi-modal contrastive learning. In our analysis, when comparing single-modal to multi-modal contrastive learning, the augmentation in the single-modal case often does not improve the signal-to-noise ratio (SNR) as effectively as having a second, complementary modality would. Therefore, in scenarios where the SNR is not significantly enhanced by augmentation, single-modal learning may perform worse than multi-modal learning.

Having said that, this does not imply that single-modal methods like SimCLR and MoCo fail in general. Empirical evidence shows that these methods are quite effective, particularly when the number of training samples is large and the SNR is sufficiently high. Our theoretical framework can indeed provide guarantees for the success of single-modal contrastive learning under such conditions.

---

References

[1] Invariant Risk Minimization arXiv 2019.

[2] Do CLIPs Always Generalize Better than ImageNet Models? arXiv 2024.