Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variables

We thank the reviewer for the valuable feedback, insightful questions, and constructive suggestions for improvement, and for appreciating our contribution to the understanding of multi-modal learning. We address weaknesses and questions as follows.

• [W1: notation] Thank you for the helpful feedback! In the revision, we will (1) add a notation table clarifying all symbols and variables; (2) ensure that important notations are adequately explained.

• [W2 and Q1: architectures and datasets] We appreciate the reviewer’s questions on alternative architectures and datasets, and we will address them as follows.

  • [architectures] Although our theory treats the function class $\mathcal{H}$ as fixed and is agnostic to the specification of $\mathcal{H}$, we may expect $\mathcal{H}$ to have “expressive power” such that the set of “ideal” representation maps is not empty. We would like to clarify that the choice in experiments (i.e. the class of 5-layer ReLU MLPs) is only for illustrative purposes to justify our theory, and it turns out to be adequately powerful for datasets such as CITE-seq. In this sense, it is very meaningful to implement more powerful architectures in simulations and applications. We have additionally tried a small-scale Transformer (2 layers, 2 heads, feedforward dimension 64, and embedding dimension $\max(100, d_x, d_y)$), topped with a final linear layer with user-specified output dimension, on CITE-seq dataset (same setting with Figure 5), the results of which are presented in the following table. From the table, we can see that both level-1 and level-2 accuracies tend to saturate as the intrinsic dimension exceeds 15, which is similar to the results presented in Figure 5. | output dimension | 1 | 7 | 13 | 19 | 25 | 31 | 37 | |---------------------|------|------|------|------|------|------|------| | ID of f(X) | 1.00 | 5.67 | 9.79 | 12.19| 14.15| 15.67| 15.97| | ID of g(Y) | 1.00 | 5.15 | 8.63 | 10.39| 11.75| 12.64| 12.96| | cell type acc (1) | 0.60 | 0.93 | 0.94 | 0.94 | 0.95 | 0.95 | 0.95 | | cell type acc (2) | 0.22 | 0.78 | 0.79 | 0.80 | 0.81 | 0.79 | 0.81 |

  • [datasets] It is meaningful to evaluate our theory on more real-world, especially heterogeneous, datasets. In addition to the CITE-seq dataset presented in the main paper, we would like to note that results on image-text datasets such as ImageNetv2 and YFCC are presented in Appendix G.4, G.6, and G.7. For these datasets, we also observe the convergence of optimized temperature to zero as is shown in Appendix G.4. In our updated version, we will present more experimental results, and it would also be interesting to quantify and compare the “intrinsic dimensions” across datasets.

• [Q2: practical guidance] Thank you for raising this insightful question. Our theoretical and experimental findings suggest that when contrastive learning is combined with temperature optimization, the learned features naturally adapt to the intrinsic dimension (ID) of the dataset. In practice, we recommend the following guideline: (1) Optimize the temperature parameter during training (as done in CLIP) to ensure that feature learning aligns with the data’s ID. (2) Start with a large output dimension and pretrain the model for a few hundred epochs to estimate the ID of the shared features. Then, iteratively (e.g., via binary search) reduce the output dimension until it is close to the estimated ID (as illustrated in Panel 4, Figure 5). This approach maintains accuracy while reducing inference cost in downstream tasks.

We thank the reviewer for the positive feedback on our work, especially the rigor of our theoretical analysis, and the comprehensiveness of our empirical validation. Below, we address the weaknesses you identified.

• [W1: assumption on $\mathcal{V}(\mathcal{H})$] Thank you for pointing this out!

  • Regarding the assumption on $\mathcal{V}(\mathcal{H})$, we agree that it is an idealized statistical setting, in which we could sharply understand the performance of multi-modal contrastive learning, and the established theory can also explain the observed phenomena, especially the convergence of temperature and the revealed intrinsic dimension, in some real-world datasets (e.g. CITE-seq and ImageNet, additional evidence is presented in Appendix G.4 and G.6).
  • We agree with the reviewer that although our theory treats the function class $\mathcal{H}$ as fixed and is agnostic to the specification of $\mathcal{H}$, we may expect $\mathcal{H}$ to have “expressive power” such that the set of “ideal” representation maps is not empty. The choice in experiments (i.e. the class of 5-layer ReLU MLPs) is only for illustrative purposes to justify our theory, and it turns out to be adequately powerful for datasets such as CITE-seq. In this sense, it is very meaningful to implement more powerful architectures in simulations and applications. We have additionally tried a small-scale Transformer (2 layers, 2 heads, feedforward dimension 64, and embedding dimension $\max(100, d_x, d_y)$), topped with a final linear layer with user-specified output dimension, on CITE-seq dataset (same setting with Figure 5), the results of which are presented in the following table. From the table, we can see that both level-1 and level-2 accuracies tend to saturate as the intrinsic dimension exceeds 15, which is similar to the results presented in Figure 5. These results indicate that our theory is agnostic to network architecture as long as the underlying architecture is sufficiently “expressive” so that the ideal representation maps can be well-approximated. | output dimension | 1 | 7 | 13 | 19 | 25 | 31 | 37 | |---------------------|------|------|------|------|------|------|------| | ID of f(X) | 1.00 | 5.67 | 9.79 | 12.19| 14.15| 15.67| 15.97| | ID of g(Y) | 1.00 | 5.15 | 8.63 | 10.39| 11.75| 12.64| 12.96| | cell type acc (1) | 0.60 | 0.93 | 0.94 | 0.94 | 0.95 | 0.95 | 0.95 | | cell type acc (2) | 0.22 | 0.78 | 0.79 | 0.80 | 0.81 | 0.79 | 0.81 | We will add a detailed discussion on this condition and its connection with architectures and datasets.

• [W2: readability of Section 3.1] Thank you for your helpful suggestions on exposition! To improve clarity and readability, we will revise Section 3.1 in the updated version to include a more intuitive and high-level explanation before presenting the formal definitions. This will highlight the core intuition behind our asymptotic analysis in the regime as $\tau$ goes to zero and better motivate the role of the approximate minimizers that we defined.

We thank the reviewer for the valuable feedback, insightful questions, constructive suggestions for improvement, and for appreciating the novelty and quality of our theoretical analysis that incorporates temperature optimization. We address weaknesses and questions as follows.

• [W1: novelty] We acknowledge that some of the studied empirical phenomena have been observed in prior works. Our contribution is to rigorously formalize and unify these observations through a statistical lens, showing how they arise naturally from contrastive learning via mutual information maximization and intrinsic dimension adaptation. Prior works reported isolated effects without fully clarifying the underlying mechanisms or their connections. Our theory provides the first principled justification and explanation of these links, bridging empirical findings with fundamental properties of contrastive methods. We welcome additional references on empirical findings to further strengthen the discussion in our revision.

• [W2: role of multi-modality] We appreciate this insightful comment. Our framework is motivated by multimodal settings because shared latent features and their intrinsic dimension are most naturally and meaningfully defined when aligning truly different modalities (e.g., RNA–protein, image–text). This is where temperature optimization and dimension adaptation are crucial, as different modalities may have heterogeneous noise levels, making it important to adaptively select the embedding dimension. That said, the framework is general enough to recover single-modality contrastive learning as a special case: under a fixed augmentation strategy, the augmented data acts as a second modality, and the intrinsic dimension is determined by the augmentation-induced shared information. This demonstrates that our results apply broadly but are particularly impactful in multi-modal integration, where shared structure is less obvious and the benefits of adaptive temperature/dimension are amplified. We will revise the paper to make this motivation explicit and highlight modality-specific implications of our theory.

• [W3: paper structure] Thank you for pointing this out! We agree that referencing Lemma 8’s variational decomposition prior to formally introducing Definition 1 may interrupt the logic flow. In the revised version, we will adjust the structure to introduce key definitions earlier and streamline cross-references where possible. Regarding Theorem 1, our intention was to present the core insight informally up front to convey intuition before delving into technical details. That said, we understand the concern, and in an updated version, we will point to the formal theorem to avoid confusion.

• [Q1: real-world practice] Thank you for raising this insightful question. Our theoretical and experimental findings suggest that when contrastive learning is combined with temperature optimization, the learned features naturally adapt to the intrinsic dimension (ID) of the dataset. In practice, we recommend the following guideline: (1) Optimize the temperature parameter during training (as done in CLIP) to ensure that feature learning aligns with the data’s ID. (2) Start with a large output dimension and pretrain the model for a few hundred epochs to estimate the ID of the shared features. Then, iteratively (e.g., via binary search) reduce the output dimension until it is close to the estimated ID (as illustrated in Panel 4, Figure 5). This approach maintains accuracy while reducing inference cost in downstream tasks.

• [Q2: capacity of function class and robustness of ID estimator] Thank you for this insightful question!

  • [MLE-based ID estimator] The estimator was proposed in [Levina and Bickle, 2004]. This estimator is not restricted to data with linear structures, indicating that it can handle data supported on nonlinear manifolds (unlike eigenvalue-based methods such as global PCA). In addition, the asymptotic properties, including bias and variance, are also presented in [Levina and Bickle, 2004, Section 3.1]. Concretely, the estimator is unbiased and the variance decreases as the number of effective “neighbors” increases, which guarantees the finite-sample performance when the sample size is sufficiently large.
  • [architecture] Although our theory treats the function class $\mathcal{H}$ as fixed and is agnostic to the specification of $\mathcal{H}$, we may expect $\mathcal{H}$ to have “expressive power” such that the set of “ideal” representation maps is not empty. We would like to clarify that the choice in experiments (i.e., the class of 5-layer ReLU MLPs) is only for illustrative purposes to justify our theory, and it turns out to be adequately powerful for datasets such as CITE-seq. In this sense, it is very meaningful to implement more powerful architectures in simulations and applications. We have additionally tried a small-scale Transformer (2 layers, 2 heads, feedforward dimension 64, and embedding dimension $\max(100, d_x,d_y)$), topped with a final linear layer with user-specified output dimension, on CITE-seq dataset, the results of which are presented in the following table. From the table, we can see that both level-1 and level-2 accuracies tend to saturate as the intrinsic dimension exceeds 15, which is similar to the results presented in Figure 5.

• [Q3: modality-specific features] Our theory focuses on shared representation adaptation under temperature optimization. Modality-specific components require specialized losses or post-processing (see [Liang et al., 2023; Wang et al., 2024]). Their presence does not contradict our results: when shared dimension is small, adapting to it is crucial, and our method facilitates subsequent extraction of unique modality features. We will clarify this point.

• [Q4: heterogeneous modalities] We agree that testing on more heterogeneous datasets is valuable. In addition to CITE-seq, we report results on image–text datasets such as ImageNetv2 and YFCC (Appendix G.4, G.6, G.7). We will expand experiments and quantify “intrinsic dimensions” across datasets in the revision.

References

• Levina, Elizaveta, and Peter Bickel. "Maximum likelihood estimation of intrinsic dimension." Advances in neural information processing systems 17 (2004).
• Liang, Paul Pu, Zihao Deng, Martin Q. Ma, James Y. Zou, Louis-Philippe Morency, and Ruslan Salakhutdinov. "Factorized contrastive learning: Going beyond multi-view redundancy." Advances in Neural Information Processing Systems 36 (2023): 32971-32998.
• Wang, Chenyu, Sharut Gupta, Xinyi Zhang, Sana Tonekaboni, Stefanie Jegelka, Tommi Jaakkola, and Caroline Uhler. "An information criterion for controlled disentanglement of multimodal data." arXiv preprint arXiv:2410.23996 (2024).

We thank the reviewer for the valuable feedback and constructive suggestions, and for appreciating our analysis of temperature optimization and the rigor of our theory that links similarity maximization and mutual information distillation. We address weaknesses and questions below.

• [finite-sample analysis: W1 & Q1] To understand the effect of sample size, from the empirical viewpoint, we conduct an ablation study in the same setting as Figure 3 (with a fixed output dimension 20) with varying sample size $N_{\rm train}$ from 2000 to 14000. We present the estimated IDs, in-sample top-1 accuracy, and out-of-sample top-1 accuracy in the following table, from which we can see that as $N_{\rm train} \geq 10000$, these quantities begin to saturate, which validates that there is indeed a lower bound of $N_{\rm train}$ to ensure good properties. To answer this question, it is also relevant to the convergence analysis of infoNCE loss in [Wang and Isola (2020), Theorem 1 (3)]. When we consider compact sets of maps and temperature (e.g., with a boundedness condition), we expect to derive the uniform law of large numbers for the finite-sample loss. In the updated version, we will add a detailed discussion on the convergence analysis below Lemma 8. | sample size | 2000 | 4000 | 6000 | 8000 | 10000 | 12000 | 14000 | |-------------------|------|------|------|------|-------|--------|--------| | ID of f(X) | 8.50 | 6.80 | 5.62 | 5.18 | 4.76 | 4.53 | 4.45 | | ID of g(Y) | 8.54 | 6.88 | 5.81 | 5.46 | 5.07 | 4.87 | 4.71 | | in-sample ACC | 0.38 | 0.75 | 0.91 | 0.95 | 0.96 | 0.96 | 0.96 | | out-of-sample ACC | 0.08 | 0.44 | 0.74 | 0.81 | 0.87 | 0.90 | 0.91 |

• [modality gap and $\mathcal{V}(\mathcal{H})$: W2 & Q2] Thank you for pointing this out and for very helpful references! We will definitely add these references and related discussions on the modality gap to our updated version. Regarding the assumption on $\mathcal{V}(\mathcal{H})$, we agree that it is an idealized statistical setting, in which we could sharply understand the performance of multi-modal contrastive learning, and the established theory can also explain the observed phenomena, especially the convergence of temperature and the revealed intrinsic dimension, in some real-world datasets (e.g. CITE-seq and ImageNet, additional evidence is presented in Appendix G.4 and G.6). Also, the modality gap is indeed a crucial phenomenon in contrastive learning. To first see this empirically, we conduct an ablation study for the setting in Figure 2, where we add entrywise iid centered Gaussian noise to the first three components of X and Y, respectively (variance is set to be $\sigma^2$ and $N_{\rm train}=10000$). Empirical studies on the modality gap defined by $|| n^{-1} \sum_{i \leq n} f(X_i) - g(Y_i) ||$ with $n$ test samples [Liang et al., 2022] in the table show the following:

  • (1) When $\sigma \leq 0.2$, both the estimated ID, the similarity maximization phenomenon, and top-K accuracies are comparable to those with $\sigma=0$, indicating that our theory can be robust when the condition is approximately met. This will imply the future work on the robustness analysis of our framework, and we will add a detailed discussion on this in an updated version.
  • (2) The computed modality gap increases with $\sigma$, and its magnitude becomes more pronounced when $\sigma > 0.4$. When $\sigma = 0$, the tiny modality gap may be an artifact of the model training process, as it depends on the number of training epochs and step size scheduling. Hence, our setting with non-empty $\mathcal{V}(\mathcal{H})$ can be viewed as a baseline scenario, and the modality gap can, in this sense, quantify the difficulty in multi-modal learning, which also motivates future theoretical work on the robustness analysis. | $\sigma$ | 0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 | |----------------------------------------------|-------|-------|-------|-------|-------|-------| | modality gap | 0.056 | 0.078 | 0.075 | 0.105 | 0.159 | 0.178 | | modality gap with self-normalized representations | 0.030 | 0.047 | 0.044 | 0.065 | 0.130 | 0.134 |

• [alternative similarities and architectures: W3 & W4 & Q3]

  • [similarities] To begin with, as shown in Figure 11 and 12, the empirical performance with cosine similarity and the similarity adopted in our paper are comparable. In addition, we would like to note that the key variational decomposition in Lemma 8, as well as the results in Lemma 10, are agnostic to the similarity measure and also apply to cosine similarity. Concretely, with a general similarity (including cosine similarity, or, equivalently, the function class maps to the unit hypersphere), as we have stated in Lemma 10 in Appendix C.2, we can also show that

    • (1) with any $(f,g) \in \mathcal{V}(\mathcal{H})$, $L(f,g,\tau)$ is non-decreasing in $\tau$ and $-L(f,g,\tau)/2$ converges to the maximum mutual information as $\tau \rightarrow 0$;
    • (2) Reversely, by Lemma 10, for any $(f,g)$ as the approximate minimizer, similar to the proof in Appendix C 2.2, we can show that $(f,g)$ must maximize both mutual information and similarity, and $L(f,g,\tau)$ is non-decreasing in $\tau$. In addition, for intrinsic dimension adaptation with other similarities, specific technical tools may be needed. As a concrete example, with the cosine similarity, tools from potential minimization and geodesic projections on a hypersphere could be promising to obtain results similar to those in Appendix D.
    • To sum up, the current similarity measure is a choice that has both comparable empirical performance with cosine similarity and elegant theoretical guarantees. We will add a more detailed discussion on variants of similarity measures in the revision.

  • [architectures] Although our theory treats the function class $\mathcal{H}$ as fixed and is agnostic to the specification of $\mathcal{H}$, we expect $\mathcal{H}$ to have “expressive power” such that the set of “ideal” representation maps is not empty. We would like to clarify that the choice in experiments (i.e. the class of 5-layer ReLU MLPs) is only for illustrative purposes to justify our theory, and it turns out to be adequately powerful for datasets such as CITE-seq. In this sense, it is very meaningful to implement more powerful architectures in simulations and applications. We have additionally tried a small-scale Transformer (2 layers, 2 heads, feedforward dimension 64, and embedding dimension $\max(100, d_x, d_y)$), topped with a final linear layer with user-specified output dimension, on CITE-seq dataset (same setting with Figure 5), the results of which are presented in the following table. From the table, we can see that both level-1 and level-2 accuracies tend to saturate as the intrinsic dimension exceeds 15, which is similar to the results presented in Figure 5. These results indicate that our theory is agnostic to network architecture as long as the underlying architecture is sufficiently “expressive” so that the ideal representation maps can be well-approximated. | output dimension | 1 | 7 | 13 | 19 | 25 | 31 | 37 | |---------------------|------|------|------|------|------|------|------| | ID of f(X) | 1.00 | 5.67 | 9.79 | 12.19| 14.15| 15.67| 15.97| | ID of g(Y) | 1.00 | 5.15 | 8.63 | 10.39| 11.75| 12.64| 12.96| | cell type acc (1) | 0.60 | 0.93 | 0.94 | 0.94 | 0.95 | 0.95 | 0.95 | | cell type acc (2) | 0.22 | 0.78 | 0.79 | 0.80 | 0.81 | 0.79 | 0.81 |

• [imperfect pairing: W5 & Q4] We agree that extending our results to settings with noisy pairs, one-to-many pairings, or multi-view data is an interesting and important future direction.

  • For the noisy pairing, the simple simulation in response to the modality gap may offer us some useful insight: (1) Within a moderate range of perturbations of matched features, contrastive learning tend to be robust in revealing the ideal shared features; (2) When the perturbation is large, the modality gap in learned representations cannot be ignored, in which case, either new algorithms or advanced technical tools ought to be designed. Moreover, noisy pairing may also be related to multi-modal data with shared and modality-specific features mentioned by Reviewer dKqv.
  • For the one-to-many scenario (e.g., for the pair $(X,Y)=({\rm image}, {\rm caption})$, suppose there are additional/augmented pairs taking the form $(X,Y_{\rm aug})$ in the training set [Waseda et al., 2025]), if our goal is still to identify shared features and to ensure that our theory applies, one shortcut solution is to predefine an integration map $T_y$ (e.g., early fusion for the text modality), and define $T_y(Y,Y_{\rm aug})$ as the new inputs. Then, if the augmentation maps are invertible (e.g., permutations), if $V(H) \neq \varnothing$ is true for each (image, caption) pair, then it can still hold for $(X, T_y(Y,Y_{\rm aug}))$ in this simple setting. If more general scenarios with one-to-many and many-to-one pairs are of interest, more sophisticated statistical models should be introduced and a fine-grained set $\mathcal{V}(\mathcal{H})$ should be considered. It will be a compelling future direction to theoretically derive the theoretical guarantee in this setting to better understand the advantage of one-to-many data/augmentations in multi-modal learning. We will add detailed discussions on this aspect in the revision.

References

• Liang, Victor Weixin, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y. Zou. "Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning."
• Waseda, Futa Kai, Antonio Tejero-de-Pablos, and Isao Echizen. "Leveraging One-To-Many Relationships in Multimodal Adversarial Defense for Robust Image-Text Retrieval."